02.12.2021 17:25:00,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 83 02.12.2021 17:25:00,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:00,013 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59916ms for 60000ms of delay, sleeping for 94ms 02.12.2021 17:25:00,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 171 02.12.2021 17:25:00,024 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:00,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,093 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 267 02.12.2021 17:25:00,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:00,110 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59868ms for 60000ms of delay, sleeping for 142ms 02.12.2021 17:25:00,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 69 02.12.2021 17:25:00,124 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:00,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:00,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,189 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 172 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 55 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 152034ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 152034ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:00,191 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,191 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:00,192 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:00,192 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:00,192 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 32, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:00,193 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1301) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:00,194 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975425, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:00,194 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975435, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:00,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:00,195 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:00,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,195 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:00,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 166 02.12.2021 17:25:00,201 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239967ms for 240000ms of delay, sleeping for 43ms 02.12.2021 17:25:00,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:00,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:00,250 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239898ms for 240000ms of delay, sleeping for 112ms 02.12.2021 17:25:00,255 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:00,256 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59865ms for 60000ms of delay, sleeping for 145ms 02.12.2021 17:25:00,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 72 02.12.2021 17:25:00,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 66 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,362 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,362 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,363 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,363 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 185 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,363 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=663) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:00,363 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=185,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:00,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:00,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 232 02.12.2021 17:25:00,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 327 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@5d7cb3c0 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@31eed1c6 02.12.2021 17:25:00,364 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:00,364 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,364 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:00,365 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,364 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 185 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:00,365 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:00,365 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:00,365 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 81, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:00,365 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=664) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 230 02.12.2021 17:25:00,367 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239511ms for 240000ms of delay, sleeping for 499ms 02.12.2021 17:25:00,365 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=664) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:00,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 323 02.12.2021 17:25:00,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:00,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 9 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:00,409 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59628ms for 60000ms of delay, sleeping for 382ms 02.12.2021 17:25:00,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:00,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,464 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,463 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:00,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:00,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,476 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:00,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,519 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:00,519 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:00,520 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=947) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:00,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:00,596 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:00,598 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,620 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=947): org.apache.kafka.common.requests.HeartbeatResponse@5b0176df 02.12.2021 17:25:00,620 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:00,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:00,693 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:00,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,701 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,706 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:00,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:00,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:00,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 1 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:00,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:00,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:00,797 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147722, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:00,797 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147732, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:00,798 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:00,804 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:00,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,805 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:00,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:00,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:00,809 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,810 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59036ms for 60000ms of delay, sleeping for 974ms 02.12.2021 17:25:00,809 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,820 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,829 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:00,867 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238508ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:00,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:00,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:00,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:00,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:00,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:00,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:01,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,010 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:01,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:01,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:01,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:01,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,218 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,219 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:01,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,321 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:01,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:01,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,443 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:01,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:01,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:01,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:01,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:01,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:01,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:01,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,606 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:01,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,723 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:01,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:01,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:01,792 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58875ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:01,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:01,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,838 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 484 02.12.2021 17:25:01,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,843 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 461 02.12.2021 17:25:01,867 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239508ms delay out of 240000ms. Remaining 492ms. Executor queue at 0 02.12.2021 17:25:01,877 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238961ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:01,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 384 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:01,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:01,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:01,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:01,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:01,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,013 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:02,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:02,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 257 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,108 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:02,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 157 02.12.2021 17:25:02,227 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:02,227 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,228 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:02,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 56 02.12.2021 17:25:02,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,305 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,305 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,305 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=646) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:02,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 38 02.12.2021 17:25:02,306 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=646): org.apache.kafka.common.requests.HeartbeatResponse@5634c8a5 02.12.2021 17:25:02,306 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 37 02.12.2021 17:25:02,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 37 02.12.2021 17:25:02,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:02,323 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:02,323 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:02,324 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=646) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:02,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 16 02.12.2021 17:25:02,324 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=646): org.apache.kafka.common.requests.HeartbeatResponse@2db47f5f 02.12.2021 17:25:02,324 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:02,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 15 02.12.2021 17:25:02,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 15 02.12.2021 17:25:02,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,340 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,344 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,440 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:02,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:02,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:02,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:02,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,540 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:02,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:02,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:02,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,609 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,609 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,611 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:02,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:02,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:02,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,745 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,746 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:02,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:02,792 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59875ms delay out of 60000ms. Remaining 125ms. Executor queue at 0 02.12.2021 17:25:02,793 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58771ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:02,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:02,845 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,846 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:02,882 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239966ms delay out of 240000ms. Remaining 34ms. Executor queue at 1 02.12.2021 17:25:02,883 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239911ms for 240000ms of delay, sleeping for 99ms 02.12.2021 17:25:02,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:02,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:02,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:02,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:02,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:02,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:02,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:02,984 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239344ms for 240000ms of delay, sleeping for 666ms 02.12.2021 17:25:03,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:03,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:03,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:03,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:03,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:03,249 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,253 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,346 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,350 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,351 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,529 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:03,530 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:03,530 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=948) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:03,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:03,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,631 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=948): org.apache.kafka.common.requests.HeartbeatResponse@1da9af70 02.12.2021 17:25:03,631 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:03,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:03,654 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239729ms for 240000ms of delay, sleeping for 281ms 02.12.2021 17:25:03,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:03,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:03,763 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:03,770 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:03,793 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59771ms delay out of 60000ms. Remaining 229ms. Executor queue at 0 02.12.2021 17:25:03,798 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 57852ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:03,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,852 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,852 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:03,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:03,887 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=656): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:03,887 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:03,887 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,887 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:03,887 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=674): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:03,895 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:03,896 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,896 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:03,896 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:03,896 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=675) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:03,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:03,895 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:03,897 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=657) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:03,902 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 50 02.12.2021 17:25:03,940 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239988ms for 240000ms of delay, sleeping for 22ms 02.12.2021 17:25:03,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:03,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:03,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=675): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,962 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:03,962 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:03,962 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:03,962 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:03,962 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:03,963 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:03,963 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:03,963 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:03,963 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:03,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 23 02.12.2021 17:25:03,963 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=657): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,966 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,967 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:03,967 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:03,967 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:03,967 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:03,967 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:03,968 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:03,968 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:03,968 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:03,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 491 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 155817ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 57 iterations. 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 57 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 155817ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:03,974 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1301): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:03,975 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=55, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:03,975 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:03,975 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:03,975 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:03,975 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975435, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:03,975 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975445, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:03,975 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1302) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:03,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:03,976 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:03,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,976 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:03,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 57 iterations. 02.12.2021 17:25:03,976 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239136ms for 240000ms of delay, sleeping for 874ms 02.12.2021 17:25:03,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:03,977 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1302): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,981 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:03,982 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=55, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:03,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:03,990 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:03,990 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:03,990 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:03,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:03,992 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:03,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:03,992 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:03,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:03,994 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:03,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:03,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:03,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:03,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:03,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:03,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:03,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:03,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:03,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,020 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,020 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:04,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 389 02.12.2021 17:25:04,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:04,096 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:04,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:04,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,122 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,122 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,123 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,153 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:04,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 285 02.12.2021 17:25:04,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:04,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:04,253 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:04,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 184 02.12.2021 17:25:04,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:04,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 515 ms and fetched 0 records 02.12.2021 17:25:04,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,289 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,289 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 83 02.12.2021 17:25:04,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:04,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,375 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:04,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:04,457 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:04,457 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:04,458 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,466 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,521 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:04,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:04,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:04,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:04,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:04,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:04,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:04,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,659 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:04,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:04,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:04,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 292 02.12.2021 17:25:04,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:04,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:04,784 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:04,801 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 58855ms delay out of 60000ms. Remaining 1145ms. Executor queue at 0 02.12.2021 17:25:04,791 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:04,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,802 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:04,808 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58729ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:04,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:04,851 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239686ms for 240000ms of delay, sleeping for 324ms 02.12.2021 17:25:04,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,870 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,870 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 435 02.12.2021 17:25:04,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 433 02.12.2021 17:25:04,908 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 89 02.12.2021 17:25:04,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:04,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:04,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:04,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:04,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:04,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:04,970 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 335 02.12.2021 17:25:04,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 333 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:04,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:04,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,022 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:05,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,059 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1028): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,060 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,060 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1029) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=664): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=664): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,061 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 463 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=186, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:05,061 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=665) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,062 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:05,062 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:05,062 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 248 02.12.2021 17:25:05,062 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:05,062 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:05,062 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=665) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:05,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 61 02.12.2021 17:25:05,071 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 234 02.12.2021 17:25:05,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 233 02.12.2021 17:25:05,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=665): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=665): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1029): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:05,109 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:05,108 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:05,108 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:05,109 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=186, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:05,110 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,110 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,109 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,109 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:05,110 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,110 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:05,113 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,113 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,113 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:05,113 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 414 02.12.2021 17:25:05,113 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,113 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:05,114 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:05,114 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,114 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,114 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,114 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 16 02.12.2021 17:25:05,114 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:05,115 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:05,116 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:05,116 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:05,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 202 02.12.2021 17:25:05,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,132 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:05,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 134 02.12.2021 17:25:05,177 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239969ms for 240000ms of delay, sleeping for 41ms 02.12.2021 17:25:05,192 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:05,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:05,214 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 309 02.12.2021 17:25:05,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:05,225 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239406ms for 240000ms of delay, sleeping for 604ms 02.12.2021 17:25:05,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:05,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:05,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 29 02.12.2021 17:25:05,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:05,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,305 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,305 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,305 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=647) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:05,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 65 02.12.2021 17:25:05,306 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=647): org.apache.kafka.common.requests.HeartbeatResponse@22508218 02.12.2021 17:25:05,306 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:05,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,312 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,312 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,313 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 208 02.12.2021 17:25:05,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:05,323 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:05,323 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:05,324 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=647) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:05,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 67 02.12.2021 17:25:05,324 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=647): org.apache.kafka.common.requests.HeartbeatResponse@7094c60b 02.12.2021 17:25:05,324 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:05,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 66 02.12.2021 17:25:05,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 66 02.12.2021 17:25:05,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:05,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:05,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,370 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,391 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,391 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,392 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:05,413 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 106 02.12.2021 17:25:05,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:05,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,461 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:05,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:05,498 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,514 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:05,517 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 6 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:05,523 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:05,524 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:05,524 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:05,524 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:05,524 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:05,524 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:05,525 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1030) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:05,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:05,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:05,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:05,594 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:05,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:05,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:05,617 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:05,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,634 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,634 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:05,674 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:05,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:05,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:05,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:05,735 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:05,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:05,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:05,809 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59730ms delay out of 60000ms. Remaining 270ms. Executor queue at 1 02.12.2021 17:25:05,809 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59241ms for 60000ms of delay, sleeping for 769ms 02.12.2021 17:25:05,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:05,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:05,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,822 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,830 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:05,831 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239673ms for 240000ms of delay, sleeping for 337ms 02.12.2021 17:25:05,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:05,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:05,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:05,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:05,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:05,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:05,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:05,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:05,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:05,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:05,970 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:05,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:05,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,025 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:06,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:06,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:06,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:06,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:06,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:06,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:06,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:06,173 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239946ms for 240000ms of delay, sleeping for 64ms 02.12.2021 17:25:06,186 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:06,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:06,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:06,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:06,240 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239821ms for 240000ms of delay, sleeping for 189ms 02.12.2021 17:25:06,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:06,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:06,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:06,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:06,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 295 02.12.2021 17:25:06,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:06,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:06,429 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239525ms for 240000ms of delay, sleeping for 485ms 02.12.2021 17:25:06,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:06,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 416 02.12.2021 17:25:06,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:06,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 388 02.12.2021 17:25:06,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:06,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,528 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:06,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:06,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,535 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,536 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:06,536 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=949) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:06,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:06,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:06,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 316 02.12.2021 17:25:06,580 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59527ms for 60000ms of delay, sleeping for 483ms 02.12.2021 17:25:06,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:06,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:06,609 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 287 02.12.2021 17:25:06,636 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=949): org.apache.kafka.common.requests.HeartbeatResponse@12c4525f 02.12.2021 17:25:06,636 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:06,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:06,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 216 02.12.2021 17:25:06,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:06,700 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:06,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:06,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,740 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:06,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 106 02.12.2021 17:25:06,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:06,809 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:06,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 82 02.12.2021 17:25:06,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:06,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:06,845 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:06,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 3 02.12.2021 17:25:06,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,891 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=658) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:06,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,892 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@5370260c 02.12.2021 17:25:06,891 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,893 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:06,894 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:06,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,895 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:06,895 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:06,895 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,896 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=659) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:06,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:06,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,906 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:06,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,896 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,907 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,907 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=676) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:06,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:06,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,907 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=676): org.apache.kafka.common.requests.HeartbeatResponse@3c91289f 02.12.2021 17:25:06,908 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:06,908 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:06,908 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:06,908 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:06,908 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,908 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=677) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:06,908 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 100 02.12.2021 17:25:06,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:06,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:06,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:06,909 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 57 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 158759ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 58 iterations. 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 158759ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:06,916 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c 02.12.2021 17:25:06,917 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:06,917 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-1], userDataSize=48) 02.12.2021 17:25:06,919 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:06,920 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Skip resuming since state is RUNNING 02.12.2021 17:25:06,920 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:06,921 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:06,921 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:06,921 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:06,921 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 59 02.12.2021 17:25:06,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:06,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,981 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 55 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,981 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1303) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=55,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:06,982 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 435 02.12.2021 17:25:06,982 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1303): org.apache.kafka.common.requests.HeartbeatResponse@2e9b9458 02.12.2021 17:25:06,982 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:06,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:06,983 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 55 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c 02.12.2021 17:25:06,983 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:06,984 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:06,984 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 33, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:06,984 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1304) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:06,984 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:06,984 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 432 02.12.2021 17:25:06,992 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,007 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 387 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,010 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,010 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,031 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:07,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:07,066 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59143ms for 60000ms of delay, sleeping for 867ms 02.12.2021 17:25:07,082 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:07,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 287 02.12.2021 17:25:07,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,144 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,148 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:07,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:07,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:07,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:07,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:07,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:07,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:07,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,409 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,411 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:07,417 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:07,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:07,420 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:07,420 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:07,422 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:07,422 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:07,422 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:07,423 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:07,423 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 58 iterations. 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 159266ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:07,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:07,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,428 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975445, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:07,429 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975455, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:07,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:07,429 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:07,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,429 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 58 iterations. 02.12.2021 17:25:07,434 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239320ms for 240000ms of delay, sleeping for 690ms 02.12.2021 17:25:07,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:07,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:07,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:07,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:07,532 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:07,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:07,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:07,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:07,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:07,545 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:07,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:07,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:07,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:07,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:07,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:07,649 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 409 02.12.2021 17:25:07,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:07,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:07,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 308 02.12.2021 17:25:07,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:07,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:07,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 208 02.12.2021 17:25:07,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:07,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:07,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,900 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,900 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 401 02.12.2021 17:25:07,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:07,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:07,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:07,959 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58904ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:07,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 103 02.12.2021 17:25:07,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:07,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:07,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:07,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:07,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,005 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 301 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,024 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:08,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,036 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:08,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,037 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,061 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,063 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=666) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:08,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:08,064 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,064 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 186 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,064 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=666) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=186,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:08,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 305 02.12.2021 17:25:08,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:08,065 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=666): org.apache.kafka.common.requests.HeartbeatResponse@30cb12f4 02.12.2021 17:25:08,065 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,065 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:08,065 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=667) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:08,065 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=666): org.apache.kafka.common.requests.HeartbeatResponse@55d239e7 02.12.2021 17:25:08,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 186 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:08,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:08,066 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 82, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:08,067 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=667) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:08,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 301 02.12.2021 17:25:08,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:08,126 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:08,126 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239989ms for 240000ms of delay, sleeping for 21ms 02.12.2021 17:25:08,152 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239918ms for 240000ms of delay, sleeping for 92ms 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,165 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,165 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:08,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:08,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:08,254 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239941ms for 240000ms of delay, sleeping for 69ms 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,307 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,309 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,309 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,309 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=648) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:08,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:08,309 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=648): org.apache.kafka.common.requests.HeartbeatResponse@6d994a6a 02.12.2021 17:25:08,310 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:08,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:08,324 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 236421ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:08,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:08,324 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:08,324 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:08,325 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=648) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:08,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 100 02.12.2021 17:25:08,325 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=648): org.apache.kafka.common.requests.HeartbeatResponse@3a975b42 02.12.2021 17:25:08,325 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:08,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:08,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:08,370 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:08,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,371 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,371 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30421ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:08,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,498 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:08,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:08,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:08,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:08,537 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,540 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,540 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:08,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:08,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:08,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:08,810 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:08,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:08,876 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:08,911 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:08,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30450ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:08,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:08,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:08,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:08,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,959 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59904ms delay out of 60000ms. Remaining 96ms. Executor queue at 0 02.12.2021 17:25:08,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:08,960 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 0 fetched records at offset FetchPosition{offset=1147732, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:1199] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Added READ_UNCOMMITTED fetch request for partition redacted.Cloud.Message.Callback.Retry1-5 at position FetchPosition{offset=1147732, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} to node kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.clients.FetchSessionHandler:259] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Built incremental fetch (sessionId=1002522419, epoch=216) for node 3. Added (), altered (redacted.Cloud.Message.Callback.Retry1-5), removed () out of (OSS.STAGING.Cloud.Message.Callback.Retry1-5) 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:266] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending READ_UNCOMMITTED IncrementalFetchRequest(toSend=(redacted.Cloud.Message.Callback.Retry1-5), toForget=(), implied=()) to broker kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) 02.12.2021 17:25:08,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:08,960 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending FETCH request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=950) and timeout 30000 to node 3: {replica_id=-1,max_wait_ms=500,min_bytes=1,max_bytes=52428800,isolation_level=0,session_id=1002522419,session_epoch=216,topics=[{topic=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition=5,current_leader_epoch=-1,fetch_offset=1147732,log_start_offset=-1,partition_max_bytes=1048576}]}],forgotten_topics_data=[],rack_id=} 02.12.2021 17:25:08,961 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry1-5 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:08,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 499 02.12.2021 17:25:08,962 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1180] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Skipping fetch for partition redacted.Cloud.Message.Callback.Retry1-5 because previous request to kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null) has not been processed 02.12.2021 17:25:08,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 498 02.12.2021 17:25:08,962 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received FETCH response from node 3 for request with header RequestHeader(apiKey=FETCH, apiVersion=11, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=950): org.apache.kafka.common.requests.FetchResponse@62810aec 02.12.2021 17:25:08,962 DEBUG [org.apache.kafka.clients.FetchSessionHandler:463] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Node 3 sent an incremental fetch response with throttleTimeMs = 0 for session 1002522419 with response=(redacted.Cloud.Message.Callback.Retry1-5) 02.12.2021 17:25:08,962 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:315] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Fetch READ_UNCOMMITTED at offset 1147732 for partition redacted.Cloud.Message.Callback.Retry1-5 returned fetch data (error=NONE, highWaterMark=1147916, lastStableOffset = 1147916, logStartOffset = 0, preferredReadReplica = absent, abortedTransactions = null, divergingEpoch =Optional.empty, recordsSizeInBytes=83958) 02.12.2021 17:25:08,962 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1251] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Preparing to read 83958 bytes of data for partition redacted.Cloud.Message.Callback.Retry1-5 with offset FetchPosition{offset=1147732, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 3 rack: null)], epoch=absent}} 02.12.2021 17:25:08,962 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1275] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating high watermark for partition redacted.Cloud.Message.Callback.Retry1-5 to 1147916 02.12.2021 17:25:08,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1280] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating log start offset for partition redacted.Cloud.Message.Callback.Retry1-5 to 0 02.12.2021 17:25:08,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:1285] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Updating last stable offset for partition redacted.Cloud.Message.Callback.Retry1-5 to 1147916 02.12.2021 17:25:08,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147732, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:08,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147742, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:08,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 3 ms and fetched 10 records 02.12.2021 17:25:08,963 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:08,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:08,963 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:08,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:08,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:08,964 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59933ms for 60000ms of delay, sleeping for 77ms 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:09,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,025 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,042 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59883ms for 60000ms of delay, sleeping for 127ms 02.12.2021 17:25:09,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,156 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,173 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59117ms for 60000ms of delay, sleeping for 893ms 02.12.2021 17:25:09,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:09,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:09,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:09,324 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 237421ms delay out of 240000ms. Remaining 2579ms. Executor queue at 0 02.12.2021 17:25:09,325 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 237417ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:09,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,380 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:09,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:09,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,527 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:09,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:09,543 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:09,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:09,543 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=951) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:09,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:09,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:09,631 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:09,643 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=951): org.apache.kafka.common.requests.HeartbeatResponse@7ae9d389 02.12.2021 17:25:09,644 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:09,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,657 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:09,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:09,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:09,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 81 02.12.2021 17:25:09,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:09,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,881 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,887 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:09,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:09,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:09,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:09,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:09,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:09,935 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:09,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:09,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:09,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:09,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:09,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:09,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,006 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:10,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:10,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,038 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:10,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,078 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59852ms for 60000ms of delay, sleeping for 158ms 02.12.2021 17:25:10,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:10,138 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:10,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,238 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,257 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58801ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:10,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:10,325 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 238417ms delay out of 240000ms. Remaining 1583ms. Executor queue at 1 02.12.2021 17:25:10,326 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238417ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:10,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:10,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30339ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,510 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:10,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:10,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30379ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:10,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:10,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:10,562 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:10,562 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:10,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:10,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,721 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:10,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:10,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:10,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:10,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,902 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:10,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:10,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:10,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:10,937 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:10,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:10,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 386 02.12.2021 17:25:11,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,013 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,013 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30842ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:11,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:11,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30880ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:11,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:11,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:11,239 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:11,268 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59812ms delay out of 60000ms. Remaining 188ms. Executor queue at 2 02.12.2021 17:25:11,272 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59135ms for 60000ms of delay, sleeping for 875ms 02.12.2021 17:25:11,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,310 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,310 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=649) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:11,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 110 02.12.2021 17:25:11,310 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=649): org.apache.kafka.common.requests.HeartbeatResponse@2a517674 02.12.2021 17:25:11,311 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 109 02.12.2021 17:25:11,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 109 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:11,326 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:11,326 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:11,327 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=649) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:11,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 112 02.12.2021 17:25:11,326 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239417ms delay out of 240000ms. Remaining 583ms. Executor queue at 3 02.12.2021 17:25:11,327 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=649): org.apache.kafka.common.requests.HeartbeatResponse@230fe75 02.12.2021 17:25:11,327 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:11,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 111 02.12.2021 17:25:11,328 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239405ms for 240000ms of delay, sleeping for 605ms 02.12.2021 17:25:11,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 110 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 6 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 9 02.12.2021 17:25:11,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:11,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,440 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31346ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,517 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31382ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:11,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:11,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:11,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30245ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:11,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:11,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:11,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:11,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:11,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,907 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30385ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:11,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:11,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 58 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 163787ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 163787ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:11,944 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975455, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:11,944 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975465, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:11,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:11,945 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:11,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,945 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:11,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 59 iterations. 02.12.2021 17:25:11,948 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238912ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:11,952 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:11,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,022 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31852ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,024 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,024 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:12,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31883ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,044 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,045 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:12,150 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59900ms for 60000ms of delay, sleeping for 110ms 02.12.2021 17:25:12,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30746ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:12,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:12,261 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59469ms for 60000ms of delay, sleeping for 541ms 02.12.2021 17:25:12,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:12,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30886ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:12,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,433 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,453 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32355ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:12,546 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32388ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:12,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:12,557 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:12,558 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:12,559 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=952) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:12,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:12,660 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=952): org.apache.kafka.common.requests.HeartbeatResponse@2a0ef585 02.12.2021 17:25:12,661 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,671 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31247ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:12,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:12,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:12,812 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147742, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:12,812 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147752, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:12,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:12,812 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:12,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,812 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:12,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:12,813 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58923ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:12,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:12,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31387ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:12,922 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:12,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:12,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:12,953 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239917ms delay out of 240000ms. Remaining 83ms. Executor queue at 0 02.12.2021 17:25:12,954 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239895ms for 240000ms of delay, sleeping for 115ms 02.12.2021 17:25:12,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:12,955 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:12,955 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32856ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,027 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32889ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,050 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,055 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,071 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239173ms for 240000ms of delay, sleeping for 837ms 02.12.2021 17:25:13,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:13,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31754ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:13,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:13,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:13,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:13,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31896ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,431 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,438 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,439 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,439 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:13,528 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33358ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33393ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,553 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:13,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:13,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:13,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:13,657 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:13,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,682 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,682 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32261ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:13,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:13,813 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59923ms delay out of 60000ms. Remaining 77ms. Executor queue at 0 02.12.2021 17:25:13,824 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59624ms for 60000ms of delay, sleeping for 386ms 02.12.2021 17:25:13,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:13,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:13,909 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239275ms for 240000ms of delay, sleeping for 735ms 02.12.2021 17:25:13,930 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1030): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:13,931 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:13,931 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,931 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:13,930 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=667): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:13,932 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:13,932 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,932 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:13,932 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,931 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,931 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=667): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:13,938 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=187, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:13,938 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:13,938 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:13,938 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=668) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,937 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1031) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 246 02.12.2021 17:25:13,937 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=668) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:13,937 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32404ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:13,939 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 132 02.12.2021 17:25:13,942 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:13,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:13,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 366 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:13,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:13,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 369 02.12.2021 17:25:14,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33861ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:14,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,032 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,044 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 266 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33894ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:14,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 268 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=668): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=668): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,062 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1031): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:14,063 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:14,063 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,063 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:14,064 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,064 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,064 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 8 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,063 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:14,066 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=187, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:14,066 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,067 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,067 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,067 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:14,068 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,068 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,068 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:14,068 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,068 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 376 02.12.2021 17:25:14,068 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,069 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:14,069 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,069 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:14,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 122 02.12.2021 17:25:14,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,072 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:14,072 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,073 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,073 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:14,073 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:14,073 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:14,074 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:14,074 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1032) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:14,074 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=677): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=659): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,109 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:14,110 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:14,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,110 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:14,110 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=660) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,110 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:14,110 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=678) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:14,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 422 02.12.2021 17:25:14,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 443 02.12.2021 17:25:14,145 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 166 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=678): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,152 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=660): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:14,152 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:14,153 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,153 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 56 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:14,153 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:14,154 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:14,154 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:14,154 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,154 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:14,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 402 02.12.2021 17:25:14,154 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:14,154 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:14,155 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:14,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 380 02.12.2021 17:25:14,158 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 168 02.12.2021 17:25:14,163 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:14,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 15 02.12.2021 17:25:14,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 270 02.12.2021 17:25:14,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,187 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,187 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,193 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,193 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,194 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32770ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,194 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,235 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59119ms for 60000ms of delay, sleeping for 891ms 02.12.2021 17:25:14,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 65 02.12.2021 17:25:14,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:14,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 277 02.12.2021 17:25:14,258 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 68 02.12.2021 17:25:14,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 166 02.12.2021 17:25:14,294 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,311 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,311 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,311 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=650) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:14,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 133 02.12.2021 17:25:14,313 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=650): org.apache.kafka.common.requests.HeartbeatResponse@5a50e27b 02.12.2021 17:25:14,313 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:14,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:14,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:14,329 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:14,329 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=650) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 128 02.12.2021 17:25:14,329 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=650): org.apache.kafka.common.requests.HeartbeatResponse@c7027eb 02.12.2021 17:25:14,329 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 128 02.12.2021 17:25:14,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 128 02.12.2021 17:25:14,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:14,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 177 02.12.2021 17:25:14,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 65 02.12.2021 17:25:14,394 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:14,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 30 02.12.2021 17:25:14,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 27 02.12.2021 17:25:14,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,441 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,445 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,446 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,445 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,448 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,448 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:14,449 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,461 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 76 02.12.2021 17:25:14,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:14,460 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,466 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,466 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,466 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32936ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:14,537 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,538 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34368ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:14,560 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:14,560 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:14,560 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:14,561 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:14,561 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34401ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:14,561 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:14,561 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:14,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:14,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:14,575 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:14,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:14,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,646 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239674ms for 240000ms of delay, sleeping for 336ms 02.12.2021 17:25:14,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:14,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:14,661 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:14,687 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 289 02.12.2021 17:25:14,697 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,702 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:14,752 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:14,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:14,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:14,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 189 02.12.2021 17:25:14,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:14,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:14,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:14,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:14,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:14,892 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 84 02.12.2021 17:25:14,908 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:14,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:14,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:14,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,948 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,962 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:14,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:14,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:14,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:14,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:14,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:14,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:14,983 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239949ms for 240000ms of delay, sleeping for 61ms 02.12.2021 17:25:15,034 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 169 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,041 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,041 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,045 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239596ms for 240000ms of delay, sleeping for 414ms 02.12.2021 17:25:15,049 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,062 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,063 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:15,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:15,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,089 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:15,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,090 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,130 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59043ms for 60000ms of delay, sleeping for 967ms 02.12.2021 17:25:15,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 68 02.12.2021 17:25:15,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:15,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:15,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:15,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:15,250 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:15,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:15,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:15,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:15,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:15,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:15,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:15,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:15,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:15,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:15,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:15,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:15,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:15,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 59 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 167309ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 167309ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1304): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:15,466 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=56, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:15,467 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:15,467 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:15,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,467 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:15,470 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975465, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:15,470 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1305) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:15,470 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975475, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:15,471 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1305): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:15,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 6 ms and fetched 10 records 02.12.2021 17:25:15,472 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:15,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,473 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 60 iterations. 02.12.2021 17:25:15,474 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239860ms for 240000ms of delay, sleeping for 150ms 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,557 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:15,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:15,573 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=56, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:15,577 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:15,577 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:15,577 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=953) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:15,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,590 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:15,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:15,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:15,591 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:15,592 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:15,643 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:15,650 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239446ms for 240000ms of delay, sleeping for 564ms 02.12.2021 17:25:15,657 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:15,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:15,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:15,681 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=953): org.apache.kafka.common.requests.HeartbeatResponse@7a0c1d9b 02.12.2021 17:25:15,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,683 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:15,704 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:15,705 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,705 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,711 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,712 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:15,766 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:15,768 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:15,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:15,784 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:15,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:15,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:15,866 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:15,868 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:15,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:15,895 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:15,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:15,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:15,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:15,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:15,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,973 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:15,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:15,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:15,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:15,987 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:15,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:15,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:15,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:16,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:16,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,099 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59037ms for 60000ms of delay, sleeping for 973ms 02.12.2021 17:25:16,095 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:16,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:16,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:16,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:16,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:16,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:16,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,215 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,228 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239397ms for 240000ms of delay, sleeping for 613ms 02.12.2021 17:25:16,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:16,276 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:16,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 291 02.12.2021 17:25:16,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:16,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:16,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:16,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:16,377 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:16,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 189 02.12.2021 17:25:16,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:16,410 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:16,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:16,467 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,479 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:16,488 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:16,490 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,494 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,497 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 427 02.12.2021 17:25:16,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:16,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,565 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:16,571 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:16,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:16,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:16,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 322 02.12.2021 17:25:16,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:16,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:16,671 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:16,695 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:16,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 221 02.12.2021 17:25:16,715 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,716 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,716 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,719 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 219 02.12.2021 17:25:16,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:16,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:16,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:16,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 120 02.12.2021 17:25:16,809 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:16,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 119 02.12.2021 17:25:16,847 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239898ms for 240000ms of delay, sleeping for 112ms 02.12.2021 17:25:16,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:16,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:16,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:16,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 20 02.12.2021 17:25:16,919 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:16,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 17 02.12.2021 17:25:16,933 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:16,936 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,936 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=669) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:16,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 69 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 187 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=669) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=187,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:16,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:16,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 280 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=669): org.apache.kafka.common.requests.HeartbeatResponse@22560ee9 02.12.2021 17:25:16,939 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:16,939 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:16,940 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,939 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=669): org.apache.kafka.common.requests.HeartbeatResponse@650d5f8f 02.12.2021 17:25:16,943 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=670) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:16,943 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:16,943 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 187 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:16,943 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 83, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:16,944 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=670) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:16,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 61 02.12.2021 17:25:16,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 276 02.12.2021 17:25:16,962 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239857ms for 240000ms of delay, sleeping for 153ms 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:16,971 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:16,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 340 02.12.2021 17:25:16,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:16,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:16,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:16,995 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 334 02.12.2021 17:25:17,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,005 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,006 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,006 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,036 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,039 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:17,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 42 02.12.2021 17:25:17,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,069 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 40 02.12.2021 17:25:17,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 239 02.12.2021 17:25:17,074 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59978ms for 60000ms of delay, sleeping for 32ms 02.12.2021 17:25:17,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 234 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,103 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,108 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59281ms for 60000ms of delay, sleeping for 729ms 02.12.2021 17:25:17,109 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,109 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,109 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=661) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:17,110 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,110 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 56 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,110 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=679) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=56,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:17,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 460 02.12.2021 17:25:17,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 458 02.12.2021 17:25:17,111 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@42c60146 02.12.2021 17:25:17,111 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,111 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:17,111 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,111 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,111 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,111 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=662) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 456 02.12.2021 17:25:17,110 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=679): org.apache.kafka.common.requests.HeartbeatResponse@25a4d51b 02.12.2021 17:25:17,112 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:17,112 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 56 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:17,112 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:17,112 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:17,112 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:17,112 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=680) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:17,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 458 02.12.2021 17:25:17,117 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239167ms for 240000ms of delay, sleeping for 843ms 02.12.2021 17:25:17,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 139 02.12.2021 17:25:17,196 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 133 02.12.2021 17:25:17,210 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,210 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:17,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,221 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:17,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 31 02.12.2021 17:25:17,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,312 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,313 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,314 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=651) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:17,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 159 02.12.2021 17:25:17,314 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=651): org.apache.kafka.common.requests.HeartbeatResponse@7c36eb6b 02.12.2021 17:25:17,314 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 157 02.12.2021 17:25:17,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 157 02.12.2021 17:25:17,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:17,330 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:17,330 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:17,330 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=651) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:17,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 165 02.12.2021 17:25:17,330 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=651): org.apache.kafka.common.requests.HeartbeatResponse@5a820cf0 02.12.2021 17:25:17,331 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:17,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:17,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 164 02.12.2021 17:25:17,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 57 02.12.2021 17:25:17,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 64 02.12.2021 17:25:17,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,495 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,496 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,496 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:17,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,507 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,571 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,571 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:17,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:17,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:17,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:17,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:17,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:17,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:17,843 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:17,843 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147752, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:17,843 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147762, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:17,844 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:17,844 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59980ms for 60000ms of delay, sleeping for 30ms 02.12.2021 17:25:17,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:17,889 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59501ms for 60000ms of delay, sleeping for 509ms 02.12.2021 17:25:17,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:17,968 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239780ms for 240000ms of delay, sleeping for 230ms 02.12.2021 17:25:17,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 517 ms and fetched 0 records 02.12.2021 17:25:17,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,994 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:17,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:17,996 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:17,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:17,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:17,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:17,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,010 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,015 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,079 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,079 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,080 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 393 02.12.2021 17:25:18,108 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 392 02.12.2021 17:25:18,200 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239782ms for 240000ms of delay, sleeping for 228ms 02.12.2021 17:25:18,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:18,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:18,228 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,230 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:18,309 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:18,399 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59454ms for 60000ms of delay, sleeping for 556ms 02.12.2021 17:25:18,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:18,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 60 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 170272ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 170272ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,429 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:18,429 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:18,429 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 34, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1306) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:18,430 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975475, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:18,430 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975485, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:18,430 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 61 iterations. 02.12.2021 17:25:18,431 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239278ms for 240000ms of delay, sleeping for 732ms 02.12.2021 17:25:18,493 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:18,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:18,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:18,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:18,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,582 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,582 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:18,600 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:18,600 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:18,600 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=954) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:18,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:18,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:18,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:18,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:18,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:18,703 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=954): org.apache.kafka.common.requests.HeartbeatResponse@4aeaee2f 02.12.2021 17:25:18,716 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:18,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:18,732 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:18,732 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:18,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:18,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:18,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:18,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:18,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:18,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:18,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:18,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:18,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:18,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 85 02.12.2021 17:25:18,957 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59994ms for 60000ms of delay, sleeping for 16ms 02.12.2021 17:25:18,975 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59973ms for 60000ms of delay, sleeping for 37ms 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,014 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59949ms for 60000ms of delay, sleeping for 61ms 02.12.2021 17:25:19,016 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,079 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59574ms for 60000ms of delay, sleeping for 436ms 02.12.2021 17:25:19,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:19,104 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,107 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,180 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239984ms for 240000ms of delay, sleeping for 26ms 02.12.2021 17:25:19,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:19,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:19,211 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239100ms for 240000ms of delay, sleeping for 910ms 02.12.2021 17:25:19,239 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,240 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:19,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:19,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:19,409 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:19,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,503 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:19,517 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59308ms for 60000ms of delay, sleeping for 702ms 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,581 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:19,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,588 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,585 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:19,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:19,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:19,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:19,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:19,608 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:19,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,605 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:19,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:19,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:19,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:19,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:19,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:19,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:19,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:19,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:19,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 303 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 321 02.12.2021 17:25:20,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,023 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:20,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,095 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,095 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 203 02.12.2021 17:25:20,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 213 02.12.2021 17:25:20,125 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239966ms for 240000ms of delay, sleeping for 44ms 02.12.2021 17:25:20,180 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239035ms for 240000ms of delay, sleeping for 975ms 02.12.2021 17:25:20,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 102 02.12.2021 17:25:20,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 113 02.12.2021 17:25:20,223 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59186ms for 60000ms of delay, sleeping for 824ms 02.12.2021 17:25:20,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,248 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 2 02.12.2021 17:25:20,313 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,316 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,316 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=652) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:20,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:20,318 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=652): org.apache.kafka.common.requests.HeartbeatResponse@5bdb349d 02.12.2021 17:25:20,319 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:20,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:20,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 9 02.12.2021 17:25:20,331 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:20,331 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:20,331 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=652) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:20,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 178 02.12.2021 17:25:20,331 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=652): org.apache.kafka.common.requests.HeartbeatResponse@5acf11a9 02.12.2021 17:25:20,331 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:20,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 178 02.12.2021 17:25:20,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:20,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 89 02.12.2021 17:25:20,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:20,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,512 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,596 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:20,600 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:20,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:20,613 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:20,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:20,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:20,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:20,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,612 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:20,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:20,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:20,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:20,758 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:20,758 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:20,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:20,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:20,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:20,937 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,025 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,025 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,057 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,058 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147762, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:21,058 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147772, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:21,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:21,058 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:21,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,058 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:21,061 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59036ms for 60000ms of delay, sleeping for 974ms 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,101 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,102 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,115 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 61 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 173002ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 173002ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:21,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,160 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975485, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:21,160 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975495, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:21,160 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 62 iterations. 02.12.2021 17:25:21,162 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239013ms for 240000ms of delay, sleeping for 997ms 02.12.2021 17:25:21,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:21,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:21,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,260 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:21,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:21,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:21,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:21,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:21,526 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:21,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,599 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,603 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,604 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,605 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:21,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:21,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:21,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:21,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:21,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:21,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:21,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:21,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,637 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:21,637 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:21,637 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=955) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:21,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:21,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:21,745 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=955): org.apache.kafka.common.requests.HeartbeatResponse@7e2b1960 02.12.2021 17:25:21,746 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:21,765 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:21,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:21,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:21,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:21,845 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 170 02.12.2021 17:25:21,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=680): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:21,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 63 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:21,956 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=662): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:21,961 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:21,961 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:21,961 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:21,961 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:21,961 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:21,960 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:21,964 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:21,964 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:21,964 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=681) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:21,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 139 02.12.2021 17:25:21,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 142 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,035 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,040 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59988ms for 60000ms of delay, sleeping for 22ms 02.12.2021 17:25:22,078 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59106ms for 60000ms of delay, sleeping for 904ms 02.12.2021 17:25:22,090 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=663): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,090 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=681): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,091 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,091 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:22,091 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,091 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:22,091 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:22,092 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,092 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,092 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,092 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,092 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,093 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,092 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 57 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:22,093 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:22,093 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,093 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 12 02.12.2021 17:25:22,093 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,093 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,093 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:22,094 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,094 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:22,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 15 02.12.2021 17:25:22,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:22,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,105 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,106 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,106 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:22,109 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:22,110 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,110 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,110 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,110 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:22,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:22,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,110 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,121 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,129 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,163 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239545ms for 240000ms of delay, sleeping for 465ms 02.12.2021 17:25:22,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,212 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:22,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:22,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:22,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,283 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,283 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:22,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:22,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:22,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:22,411 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:22,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:22,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:22,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=670): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:22,448 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=670): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,451 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=671) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 84 02.12.2021 17:25:22,450 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,451 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,451 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=671) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 332 02.12.2021 17:25:22,450 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1032): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:22,452 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=188, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:22,452 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:22,452 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:22,452 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:22,452 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1033) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:22,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 177 02.12.2021 17:25:22,486 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=671): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,486 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=671): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1033): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:22,487 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:22,487 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=188, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:22,488 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,488 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:22,488 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,488 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,488 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:22,488 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 143 02.12.2021 17:25:22,487 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:22,488 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:22,489 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,489 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,489 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,489 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 48 02.12.2021 17:25:22,489 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:22,490 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:22,490 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:22,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:22,512 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,518 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:22,521 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,538 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,539 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 41 02.12.2021 17:25:22,590 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:22,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:22,612 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:22,612 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:22,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:22,623 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:22,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:22,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,631 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:22,632 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,633 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,633 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,636 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:22,636 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,637 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,637 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,637 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239783ms for 240000ms of delay, sleeping for 227ms 02.12.2021 17:25:22,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:22,638 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:22,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:22,640 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:22,640 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:22,641 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:22,641 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:22,641 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:22,641 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1034) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:22,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:22,689 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:22,691 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:22,707 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:22,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:22,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:22,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:22,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:22,785 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:22,788 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:22,788 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:22,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:22,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:22,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:22,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:22,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:22,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:22,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:22,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:22,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:22,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:22,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:22,867 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239623ms for 240000ms of delay, sleeping for 387ms 02.12.2021 17:25:22,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:22,910 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:22,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:22,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:22,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:22,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:22,987 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59989ms for 60000ms of delay, sleeping for 21ms 02.12.2021 17:25:22,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:23,010 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59870ms for 60000ms of delay, sleeping for 140ms 02.12.2021 17:25:23,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:23,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,026 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 305 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:23,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:23,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,110 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,111 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,113 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,113 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:23,126 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 205 02.12.2021 17:25:23,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:23,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,152 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147772, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:23,152 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147782, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:23,152 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:23,153 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:23,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,153 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,153 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:23,153 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59171ms for 60000ms of delay, sleeping for 839ms 02.12.2021 17:25:23,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:23,212 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:23,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 104 02.12.2021 17:25:23,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:23,259 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239299ms for 240000ms of delay, sleeping for 711ms 02.12.2021 17:25:23,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:23,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:23,317 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,317 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:23,317 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=653) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:23,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 202 02.12.2021 17:25:23,320 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=653): org.apache.kafka.common.requests.HeartbeatResponse@7a980b44 02.12.2021 17:25:23,320 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:23,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:23,337 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:23,337 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:23,337 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=653) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:23,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:23,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=653): org.apache.kafka.common.requests.HeartbeatResponse@5f099fd3 02.12.2021 17:25:23,338 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:23,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:23,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:23,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:23,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 395 02.12.2021 17:25:23,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:23,417 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:23,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:23,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:23,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:23,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:23,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:23,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,527 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:23,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,554 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,554 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:23,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,614 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,614 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:23,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:23,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:23,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:23,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:23,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:23,645 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:23,645 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:23,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:23,726 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:23,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:23,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:23,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:23,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:23,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:23,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:23,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:23,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 187 02.12.2021 17:25:23,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:23,855 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:23,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:23,919 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:23,928 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:23,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 73 02.12.2021 17:25:23,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:23,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:23,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238991ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:23,996 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59765ms for 60000ms of delay, sleeping for 245ms 02.12.2021 17:25:24,015 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:24,023 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:24,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,039 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 84 02.12.2021 17:25:24,065 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:24,066 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,066 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 186 02.12.2021 17:25:24,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:24,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:24,129 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:24,140 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,146 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,167 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 86 02.12.2021 17:25:24,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:24,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:24,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:24,240 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:24,244 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59198ms for 60000ms of delay, sleeping for 812ms 02.12.2021 17:25:24,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,307 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:24,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:24,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:24,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:24,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:24,408 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:24,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:24,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:24,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:24,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:24,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:24,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:24,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,532 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,532 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,533 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,539 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,567 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,609 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:24,618 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:24,618 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,618 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:24,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:24,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:24,619 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:24,619 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 342 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 337 02.12.2021 17:25:24,634 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:24,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:24,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:24,659 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:24,659 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:24,660 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=956) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:24,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:24,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 242 02.12.2021 17:25:24,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 236 02.12.2021 17:25:24,735 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:24,760 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=956): org.apache.kafka.common.requests.HeartbeatResponse@235469fd 02.12.2021 17:25:24,760 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:24,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:24,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 294 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:24,810 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:24,811 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:24,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:24,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 142 02.12.2021 17:25:24,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 136 02.12.2021 17:25:24,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:24,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 189 02.12.2021 17:25:24,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:24,911 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:24,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 41 02.12.2021 17:25:24,931 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 34 02.12.2021 17:25:24,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:24,962 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,963 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=664) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:24,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 156 02.12.2021 17:25:24,963 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@4336e194 02.12.2021 17:25:24,963 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:24,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:24,963 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:24,963 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:24,964 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:24,964 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,964 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=665) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:24,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 155 02.12.2021 17:25:24,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:24,965 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:24,965 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,965 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=682) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:24,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 162 02.12.2021 17:25:24,965 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=682): org.apache.kafka.common.requests.HeartbeatResponse@22a5084 02.12.2021 17:25:24,966 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:24,966 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 57 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:24,966 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:24,966 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:24,966 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:24,966 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=683) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:24,966 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 161 02.12.2021 17:25:24,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:24,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239991ms delay out of 240000ms. Remaining 9ms. Executor queue at 0 02.12.2021 17:25:24,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239791ms for 240000ms of delay, sleeping for 219ms 02.12.2021 17:25:25,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:25,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,032 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,050 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,051 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,058 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59944ms for 60000ms of delay, sleeping for 66ms 02.12.2021 17:25:25,063 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,065 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,068 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,068 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 383 02.12.2021 17:25:25,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:25,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,120 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,126 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59998ms for 60000ms of delay, sleeping for 12ms 02.12.2021 17:25:25,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,128 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:25,143 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59398ms for 60000ms of delay, sleeping for 612ms 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,149 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:25,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 283 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 62 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 177044ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:25,201 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 177045ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:25,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:25,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,204 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1306): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:25,206 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=57, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:25,208 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:25,209 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:25,209 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:25,210 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1307) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:25,210 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975495, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:25,217 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975505, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:25,217 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1307): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:25,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 13 ms and fetched 10 records 02.12.2021 17:25:25,217 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:25,217 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,217 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 63 iterations. 02.12.2021 17:25:25,212 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:25,231 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239908ms for 240000ms of delay, sleeping for 102ms 02.12.2021 17:25:25,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:25,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:25,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 173 02.12.2021 17:25:25,310 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=57, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 518 ms and fetched 0 records 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,329 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 122 02.12.2021 17:25:25,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:25,336 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239202ms for 240000ms of delay, sleeping for 808ms 02.12.2021 17:25:25,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:25,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 73 02.12.2021 17:25:25,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 21 02.12.2021 17:25:25,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:25,451 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,453 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,453 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=672) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:25,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:25,451 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:25,453 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 188 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,453 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=672) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=188,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:25,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:25,454 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 375 02.12.2021 17:25:25,454 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=672): org.apache.kafka.common.requests.HeartbeatResponse@2738021b 02.12.2021 17:25:25,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 117 02.12.2021 17:25:25,455 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=672): org.apache.kafka.common.requests.HeartbeatResponse@534d344d 02.12.2021 17:25:25,455 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,455 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:25,455 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,455 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,455 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=673) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 112 02.12.2021 17:25:25,454 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 188 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:25,456 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:25,456 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:25,457 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 84, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:25,457 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=673) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:25,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 371 02.12.2021 17:25:25,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,542 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,544 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,554 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,553 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:25,551 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,565 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,565 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,573 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:25,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:25,633 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:25,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:25,632 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,638 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:25,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:25,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,661 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 392 02.12.2021 17:25:25,661 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:25,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,669 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,669 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:25,767 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59824ms for 60000ms of delay, sleeping for 186ms 02.12.2021 17:25:25,767 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 286 02.12.2021 17:25:25,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:25,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:25,829 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,829 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:25,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:25,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:25,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:25,836 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:25,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:25,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:25,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:25,972 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147782, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:25,981 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147792, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:25,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 18 ms and fetched 10 records 02.12.2021 17:25:25,981 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:25,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:25,981 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:25,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:25,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:25,982 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59861ms for 60000ms of delay, sleeping for 149ms 02.12.2021 17:25:25,981 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:25,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:26,058 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 252 02.12.2021 17:25:26,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,070 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,076 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,077 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,077 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 260 02.12.2021 17:25:26,132 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59928ms for 60000ms of delay, sleeping for 82ms 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,134 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,146 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,148 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239179ms for 240000ms of delay, sleeping for 831ms 02.12.2021 17:25:26,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 151 02.12.2021 17:25:26,169 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,170 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 159 02.12.2021 17:25:26,216 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59193ms for 60000ms of delay, sleeping for 817ms 02.12.2021 17:25:26,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 48 02.12.2021 17:25:26,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 59 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,320 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,321 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,321 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,321 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=654) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:26,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:26,321 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=654): org.apache.kafka.common.requests.HeartbeatResponse@27066d56 02.12.2021 17:25:26,321 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:26,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 244 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:26,341 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:26,341 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:26,341 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=654) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:26,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 236 02.12.2021 17:25:26,341 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=654): org.apache.kafka.common.requests.HeartbeatResponse@524bb4a6 02.12.2021 17:25:26,341 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:26,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 236 02.12.2021 17:25:26,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 235 02.12.2021 17:25:26,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,349 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 143 02.12.2021 17:25:26,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 135 02.12.2021 17:25:26,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 43 02.12.2021 17:25:26,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 34 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,566 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,567 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,577 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:26,574 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,580 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,635 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:26,636 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:26,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,647 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:26,647 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:26,667 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:26,672 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:26,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:26,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:26,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:26,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:26,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:26,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:26,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:26,876 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:26,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:26,977 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:26,980 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239778ms for 240000ms of delay, sleeping for 232ms 02.12.2021 17:25:26,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:27,035 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59031ms for 60000ms of delay, sleeping for 979ms 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,067 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,078 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,078 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:27,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,086 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,086 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:27,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,142 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,150 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,150 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,176 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,178 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,177 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:27,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,216 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239517ms for 240000ms of delay, sleeping for 493ms 02.12.2021 17:25:27,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:27,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,369 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:27,380 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:27,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:27,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:27,568 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,569 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,583 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,584 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:27,586 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,587 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,643 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,643 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:27,647 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:27,648 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:27,650 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:27,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,654 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:27,682 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:27,682 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,682 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:27,682 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:27,673 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:27,670 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:27,683 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:27,684 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=957) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:27,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:27,683 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:27,690 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30218ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:27,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:27,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:27,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,710 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 237861ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:27,784 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=957): org.apache.kafka.common.requests.HeartbeatResponse@39ae58c9 02.12.2021 17:25:27,783 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 286 02.12.2021 17:25:27,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:27,788 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:27,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:27,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:27,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:27,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 181 02.12.2021 17:25:27,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:27,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:28,019 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59571ms for 60000ms of delay, sleeping for 439ms 02.12.2021 17:25:28,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,069 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,069 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,071 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,088 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:28,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,091 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,093 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,094 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,158 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,158 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:28,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:28,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,206 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,206 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:28,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Committing all active tasks [] and standby tasks [] since 30744ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,215 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:28,217 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 57 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:28,217 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1308) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=57,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:28,216 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,273 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:28,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:28,326 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1308): org.apache.kafka.common.requests.HeartbeatResponse@2e601f24 02.12.2021 17:25:28,326 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1114] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Ignoring heartbeat response with error REBALANCE_IN_PROGRESS during COMPLETING_REBALANCE state 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:28,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:28,460 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59411ms for 60000ms of delay, sleeping for 599ms 02.12.2021 17:25:28,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:28,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,572 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,593 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,593 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,595 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,594 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,597 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,597 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,602 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,609 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:28,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:28,653 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:28,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,660 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:28,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,661 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,675 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:28,672 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:28,710 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 238861ms delay out of 240000ms. Remaining 1139ms. Executor queue at 0 02.12.2021 17:25:28,711 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238748ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:28,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:28,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Committing all active tasks [] and standby tasks [] since 31258ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:28,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:28,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:28,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:28,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:28,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:28,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:28,859 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:28,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:28,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:28,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:29,017 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:29,061 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59129ms for 60000ms of delay, sleeping for 881ms 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,072 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,073 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 248 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,096 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,096 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 231 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,154 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,173 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 148 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,177 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 131 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Committing all active tasks [] and standby tasks [] since 31773ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:29,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,246 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 47 02.12.2021 17:25:29,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 30 02.12.2021 17:25:29,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,321 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,322 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,322 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=655) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:29,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 251 02.12.2021 17:25:29,322 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=655): org.apache.kafka.common.requests.HeartbeatResponse@6ca0b76e 02.12.2021 17:25:29,323 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 250 02.12.2021 17:25:29,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 250 02.12.2021 17:25:29,341 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:29,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:29,342 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=655) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:29,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:29,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 267 02.12.2021 17:25:29,344 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=655): org.apache.kafka.common.requests.HeartbeatResponse@1864c793 02.12.2021 17:25:29,346 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:29,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 264 02.12.2021 17:25:29,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 261 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,360 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,361 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,361 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 149 02.12.2021 17:25:29,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 161 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1034): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=673): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,480 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=674) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=673): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,480 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:29,480 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,479 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=189, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:29,481 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:29,481 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:29,481 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:29,481 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=674) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 116 02.12.2021 17:25:29,482 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 379 02.12.2021 17:25:29,481 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1035) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:29,483 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 263 02.12.2021 17:25:29,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 46 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=674): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1035): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,540 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=674): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:29,540 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=189, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,541 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,542 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,542 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,541 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:29,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:29,542 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:29,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:29,543 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,545 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,545 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:29,545 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:29,545 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,544 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:29,545 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:29,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 206 02.12.2021 17:25:29,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,549 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:29,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 320 02.12.2021 17:25:29,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 56 02.12.2021 17:25:29,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 58 02.12.2021 17:25:29,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,578 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,610 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,613 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:29,615 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:29,615 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,615 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,615 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,616 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,613 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,617 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,617 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:29,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:29,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 211 02.12.2021 17:25:29,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:29,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:29,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,677 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,689 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:29,681 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:29,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:29,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:29,713 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239750ms delay out of 240000ms. Remaining 250ms. Executor queue at 1 02.12.2021 17:25:29,719 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239559ms for 240000ms of delay, sleeping for 451ms 02.12.2021 17:25:29,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:29,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:29,749 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,749 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:29,750 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,750 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:29,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Committing all active tasks [] and standby tasks [] since 32282ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:29,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 105 02.12.2021 17:25:29,754 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:29,762 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:29,770 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:29,770 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:29,772 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:29,772 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:29,776 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1036) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:29,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 494 02.12.2021 17:25:29,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:29,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:29,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:29,850 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:29,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 5 02.12.2021 17:25:29,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:29,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:29,869 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:29,869 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:29,874 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:29,874 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:29,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:29,874 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:29,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:29,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:29,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:29,943 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59951ms for 60000ms of delay, sleeping for 59ms 02.12.2021 17:25:29,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:30,003 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59747ms for 60000ms of delay, sleeping for 263ms 02.12.2021 17:25:30,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 71 02.12.2021 17:25:30,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:30,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:30,075 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:30,080 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,083 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,116 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,118 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,119 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 63 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,172 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 182015ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:30,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:30,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,173 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 182016ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:30,174 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,174 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:30,175 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:30,175 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:30,175 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:30,176 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 35, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:30,180 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1309) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:30,180 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975505, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=665): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,181 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,181 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=666) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=683): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:30,182 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:30,182 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=684) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:30,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 7 02.12.2021 17:25:30,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 478 02.12.2021 17:25:30,180 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975515, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1309): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:30,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:30,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 10 ms and fetched 10 records 02.12.2021 17:25:30,186 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,187 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 64 iterations. 02.12.2021 17:25:30,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,188 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239031ms for 240000ms of delay, sleeping for 979ms 02.12.2021 17:25:30,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,190 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:30,220 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=684): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,225 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,225 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:30,225 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:30,225 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,225 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,225 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,226 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,226 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,226 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:30,226 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,226 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 473 02.12.2021 17:25:30,220 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=666): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,226 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:30,226 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:30,226 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 58 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:30,226 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:30,226 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:30,226 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:30,226 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:30,227 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:30,227 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:30,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 433 02.12.2021 17:25:30,228 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Committing all active tasks [0_5] and standby tasks [] since 32053ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Flushing record collector 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Prepared RUNNING task for committing 02.12.2021 17:25:30,269 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry1-5=OffsetAndMetadata{offset=1147792, leaderEpoch=null, metadata='AQAAAX179isw'}} to coordinator kafka-4.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:30,269 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=958) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry1,partitions=[{partition_index=5,committed_offset=1147792,committed_leader_epoch=-1,committed_metadata=AQAAAX179isw}]}]} 02.12.2021 17:25:30,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:30,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:30,289 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received OFFSET_COMMIT response from node 2147483644 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=958): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry1', partitions=[OffsetCommitResponsePartition(partitionIndex=5, errorCode=0)])]) 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Committed offset 1147792 for partition redacted.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry1-5=1147791} 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Committed all active tasks [0_5] and standby tasks [] in 0ms 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,290 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147792, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:30,290 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147802, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:30,290 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 8 iterations. 02.12.2021 17:25:30,291 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59142ms for 60000ms of delay, sleeping for 868ms 02.12.2021 17:25:30,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:30,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:30,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 372 02.12.2021 17:25:30,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 332 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,375 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:30,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:30,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:30,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 272 02.12.2021 17:25:30,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 232 02.12.2021 17:25:30,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:30,519 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:30,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:30,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 171 02.12.2021 17:25:30,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 131 02.12.2021 17:25:30,575 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,583 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,583 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,623 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:30,623 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 71 02.12.2021 17:25:30,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 29 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,661 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,662 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:30,662 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,662 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,663 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:30,663 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:30,663 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,676 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:30,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:30,699 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:30,699 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:30,699 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=959) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:30,699 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:30,700 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:30,700 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,701 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:30,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:30,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 395 02.12.2021 17:25:30,765 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:30,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:30,780 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:30,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:30,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 89 02.12.2021 17:25:30,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:30,800 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=959): org.apache.kafka.common.requests.HeartbeatResponse@24a84245 02.12.2021 17:25:30,803 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:30,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 288 02.12.2021 17:25:30,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:30,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 292 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:30,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:30,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:30,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:30,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:30,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 188 02.12.2021 17:25:30,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:30,972 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:30,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:30,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:31,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:31,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:31,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:31,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:31,084 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,084 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,104 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:31,123 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,124 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,124 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,125 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,125 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,127 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,160 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59954ms for 60000ms of delay, sleeping for 56ms 02.12.2021 17:25:31,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,163 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,164 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:31,185 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:31,199 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239315ms for 240000ms of delay, sleeping for 695ms 02.12.2021 17:25:31,202 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,223 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59131ms for 60000ms of delay, sleeping for 879ms 02.12.2021 17:25:31,225 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:31,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:31,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:31,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:31,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,290 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:31,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:31,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:31,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,385 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,389 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:31,405 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:31,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:31,433 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:31,468 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:31,492 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:31,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 390 02.12.2021 17:25:31,505 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:31,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:31,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:31,570 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:31,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,584 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,584 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,585 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,585 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 287 02.12.2021 17:25:31,606 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,627 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:31,626 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,631 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,631 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,664 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:31,673 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,674 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:31,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:31,674 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:31,685 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:31,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 186 02.12.2021 17:25:31,686 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:31,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:31,722 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:31,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:31,732 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:31,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:31,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 85 02.12.2021 17:25:31,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:31,812 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:31,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:31,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:31,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:31,832 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:31,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:31,895 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239751ms for 240000ms of delay, sleeping for 259ms 02.12.2021 17:25:31,894 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:31,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:31,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:31,903 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:31,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 288 02.12.2021 17:25:31,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:31,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:31,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 184 02.12.2021 17:25:31,991 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:32,004 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:32,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 186 02.12.2021 17:25:32,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:32,037 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:32,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 71 02.12.2021 17:25:32,097 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,105 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,105 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 216 02.12.2021 17:25:32,106 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:32,114 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59385ms for 60000ms of delay, sleeping for 625ms 02.12.2021 17:25:32,119 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 86 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,132 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,141 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,141 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 339 02.12.2021 17:25:32,140 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,142 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:32,149 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:32,179 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239163ms for 240000ms of delay, sleeping for 847ms 02.12.2021 17:25:32,205 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 116 02.12.2021 17:25:32,205 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 519 ms and fetched 0 records 02.12.2021 17:25:32,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,213 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:32,213 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,224 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,224 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:32,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,232 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,241 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,241 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:32,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:32,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 238 02.12.2021 17:25:32,313 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 8 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 85 02.12.2021 17:25:32,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,321 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,321 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,321 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=656) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:32,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:32,322 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=656): org.apache.kafka.common.requests.HeartbeatResponse@6e5d09bb 02.12.2021 17:25:32,322 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:32,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 283 02.12.2021 17:25:32,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:32,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:32,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:32,342 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:32,342 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=656) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:32,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 302 02.12.2021 17:25:32,342 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=656): org.apache.kafka.common.requests.HeartbeatResponse@3a84df6f 02.12.2021 17:25:32,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:32,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 302 02.12.2021 17:25:32,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 301 02.12.2021 17:25:32,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 130 02.12.2021 17:25:32,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,403 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,403 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,404 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 76 02.12.2021 17:25:32,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 183 02.12.2021 17:25:32,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:32,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:32,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 201 02.12.2021 17:25:32,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 30 02.12.2021 17:25:32,480 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,481 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=675) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:32,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:32,480 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:32,483 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@146abf03 02.12.2021 17:25:32,484 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:32,484 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=676) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 157 02.12.2021 17:25:32,482 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 189 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,484 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=675) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=189,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:32,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 424 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@1f62478f 02.12.2021 17:25:32,485 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:32,485 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 189 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:32,485 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:32,486 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:32,486 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 85, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:32,486 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=676) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:32,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 418 02.12.2021 17:25:32,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:32,526 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:32,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:32,544 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 100 02.12.2021 17:25:32,582 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,585 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:32,605 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,607 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:32,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,641 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,642 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,645 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:32,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:32,727 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:32,730 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,730 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:32,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:32,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:32,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:32,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 444 02.12.2021 17:25:32,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:32,741 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:32,741 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,746 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,747 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59779ms for 60000ms of delay, sleeping for 231ms 02.12.2021 17:25:32,747 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 435 02.12.2021 17:25:32,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,819 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:32,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:32,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:32,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:32,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:32,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 341 02.12.2021 17:25:32,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:32,853 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 329 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:32,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:32,905 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:32,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:32,946 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 235 02.12.2021 17:25:32,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:32,954 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 228 02.12.2021 17:25:32,979 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59726ms for 60000ms of delay, sleeping for 284ms 02.12.2021 17:25:33,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:33,027 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239873ms for 240000ms of delay, sleeping for 137ms 02.12.2021 17:25:33,046 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 135 02.12.2021 17:25:33,052 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:33,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 128 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,110 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,110 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,143 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,143 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,145 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,146 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,146 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 34 02.12.2021 17:25:33,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 27 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 64 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 185008ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,165 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,166 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 185008ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:33,166 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:33,166 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,166 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=58, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:33,167 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:33,167 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:33,167 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:33,169 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1310) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:33,169 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975515, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:33,169 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975525, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:33,169 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1310): SyncGroupResponseData(throttleTimeMs=0, errorCode=27, protocolType=null, protocolName=null, assignment=[]) 02.12.2021 17:25:33,177 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 11 ms and fetched 10 records 02.12.2021 17:25:33,178 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:33,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,178 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,178 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 65 iterations. 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,183 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,183 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=685) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:33,182 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,184 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 58 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,184 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=667) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=58,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:33,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 53 02.12.2021 17:25:33,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 63 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=685): org.apache.kafka.common.requests.HeartbeatResponse@1776fc53 02.12.2021 17:25:33,191 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:33,191 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,191 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,192 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=686) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,190 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=667): org.apache.kafka.common.requests.HeartbeatResponse@7925a1a9 02.12.2021 17:25:33,200 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:33,200 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 58 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:33,201 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:33,201 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:33,201 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:33,201 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=668) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:33,201 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 36 02.12.2021 17:25:33,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 47 02.12.2021 17:25:33,208 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239035ms for 240000ms of delay, sleeping for 975ms 02.12.2021 17:25:33,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,247 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:33,268 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59226ms for 60000ms of delay, sleeping for 784ms 02.12.2021 17:25:33,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,269 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:781] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] SyncGroup failed: The group began another rebalance. Need to re-join the group. Sent generation was Generation{generationId=58, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:33,284 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,284 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:33,312 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,332 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,333 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 280 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,406 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:33,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 177 02.12.2021 17:25:33,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:33,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 77 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,616 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,644 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,649 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,649 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,651 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:33,662 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,665 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,662 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,670 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,682 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:33,708 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:33,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:33,716 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:33,722 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=960) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:33,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,759 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:33,785 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:33,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:33,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:33,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 295 02.12.2021 17:25:33,827 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=960): org.apache.kafka.common.requests.HeartbeatResponse@26ebfa9 02.12.2021 17:25:33,829 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:33,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:33,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:33,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:33,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:33,909 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:33,909 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:33,922 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:33,986 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:34,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:34,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 8 records with 8 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,054 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 9 iterations. 02.12.2021 17:25:34,056 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59915ms for 60000ms of delay, sleeping for 95ms 02.12.2021 17:25:34,086 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,117 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,117 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 2 records with 9 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,154 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147802, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:34,154 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147812, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:34,154 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,154 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 10 iterations. 02.12.2021 17:25:34,155 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59968ms for 60000ms of delay, sleeping for 42ms 02.12.2021 17:25:34,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:34,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,181 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,181 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,182 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,184 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,189 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239037ms for 240000ms of delay, sleeping for 973ms 02.12.2021 17:25:34,200 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59958ms for 60000ms of delay, sleeping for 52ms 02.12.2021 17:25:34,218 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:34,257 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59961ms for 60000ms of delay, sleeping for 49ms 02.12.2021 17:25:34,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,261 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,261 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,262 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,285 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,288 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,288 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,308 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59320ms for 60000ms of delay, sleeping for 690ms 02.12.2021 17:25:34,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:34,410 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,413 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,422 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:34,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:34,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:34,586 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:34,624 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,625 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,629 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,685 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,688 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,688 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,692 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:34,729 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:34,768 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,770 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,789 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,796 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:34,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:34,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:34,797 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:34,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:34,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:34,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,837 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:34,836 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:34,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:34,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:34,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:34,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:34,918 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:34,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:34,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:34,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:34,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:34,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:34,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:34,941 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:34,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:35,000 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59920ms for 60000ms of delay, sleeping for 90ms 02.12.2021 17:25:35,045 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:35,091 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59398ms for 60000ms of delay, sleeping for 612ms 02.12.2021 17:25:35,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,130 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,131 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,131 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:35,178 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239943ms for 240000ms of delay, sleeping for 67ms 02.12.2021 17:25:35,189 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,198 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,203 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,203 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,204 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,206 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 136 02.12.2021 17:25:35,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:35,249 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239328ms for 240000ms of delay, sleeping for 682ms 02.12.2021 17:25:35,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,272 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 25 02.12.2021 17:25:35,317 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,318 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,321 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,325 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,325 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,325 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=657) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:35,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 306 02.12.2021 17:25:35,325 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=657): org.apache.kafka.common.requests.HeartbeatResponse@54a5ed1f 02.12.2021 17:25:35,325 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 306 02.12.2021 17:25:35,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 305 02.12.2021 17:25:35,342 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:35,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:35,342 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=657) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=657): org.apache.kafka.common.requests.HeartbeatResponse@29e038eb 02.12.2021 17:25:35,345 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:35,343 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:35,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:35,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 205 02.12.2021 17:25:35,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 261 02.12.2021 17:25:35,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 103 02.12.2021 17:25:35,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 153 02.12.2021 17:25:35,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 1 02.12.2021 17:25:35,637 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,639 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,644 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 53 02.12.2021 17:25:35,704 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59911ms for 60000ms of delay, sleeping for 99ms 02.12.2021 17:25:35,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,708 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,710 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,710 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:35,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:35,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,775 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,813 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59975ms for 60000ms of delay, sleeping for 35ms 02.12.2021 17:25:35,811 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:35,819 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:35,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:35,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 10 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:35,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,855 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 288 02.12.2021 17:25:35,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:35,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,859 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147812, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:35,859 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147822, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:35,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 5 ms and fetched 10 records 02.12.2021 17:25:35,860 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:35,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,860 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 11 iterations. 02.12.2021 17:25:35,860 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59918ms for 60000ms of delay, sleeping for 92ms 02.12.2021 17:25:35,859 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,881 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1036): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:35,881 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:35,881 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,881 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:35,882 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,882 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1037) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 479 02.12.2021 17:25:35,881 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=676): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:35,883 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:35,883 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,882 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=676): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:35,889 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=190, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:35,889 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:35,889 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:35,889 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,889 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=677) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 34 02.12.2021 17:25:35,885 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:35,893 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:35,894 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:35,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 314 02.12.2021 17:25:35,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:35,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:35,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:35,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 65 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,953 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59795ms for 60000ms of delay, sleeping for 215ms 02.12.2021 17:25:35,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 187792ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:35,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 185 02.12.2021 17:25:35,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:35,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:35,969 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 187824ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:35,981 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:472] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Rebalance failed. 02.12.2021 17:25:35,981 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 36, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:35,981 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1311) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:35,982 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975525, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:35,982 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975535, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:35,982 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:35,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 66 iterations. 02.12.2021 17:25:35,983 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239634ms for 240000ms of delay, sleeping for 376ms 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=677): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,986 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=677): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,983 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1037): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:35,986 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:35,986 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:35,986 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:35,987 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=190, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:35,986 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:35,987 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:35,987 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:35,987 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:35,987 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:35,992 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:35,992 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:35,987 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:35,992 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:35,992 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:35,992 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:35,992 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:35,992 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:35,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 222 02.12.2021 17:25:35,992 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:35,997 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:35,990 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:35,998 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:35,998 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:35,997 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:35,998 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:35,999 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:35,999 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:35,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 441 02.12.2021 17:25:35,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 375 02.12.2021 17:25:36,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:36,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:36,081 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:36,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 111 02.12.2021 17:25:36,100 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 324 02.12.2021 17:25:36,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 258 02.12.2021 17:25:36,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 512 ms and fetched 0 records 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,151 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,177 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59393ms for 60000ms of delay, sleeping for 617ms 02.12.2021 17:25:36,198 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 10 02.12.2021 17:25:36,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 221 02.12.2021 17:25:36,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 157 02.12.2021 17:25:36,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,218 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,221 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,221 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:36,221 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,218 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,222 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,221 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,225 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,226 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,225 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,228 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:36,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,228 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,234 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,234 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,252 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 515 ms and fetched 0 records 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,291 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 120 02.12.2021 17:25:36,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 56 02.12.2021 17:25:36,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:36,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,324 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,325 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,325 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:36,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:36,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:36,360 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239235ms for 240000ms of delay, sleeping for 775ms 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,362 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,362 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,363 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:36,363 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:36,363 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1038) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:36,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 20 02.12.2021 17:25:36,406 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:36,425 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:36,426 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:36,426 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,426 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:36,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:36,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:36,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:36,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:36,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:36,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:36,627 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:36,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:36,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:36,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,652 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,653 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,655 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,655 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,728 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,731 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,731 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:36,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:36,734 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,738 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:36,742 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:36,749 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=961) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:36,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,792 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,797 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59713ms for 60000ms of delay, sleeping for 297ms 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:36,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:36,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:36,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:36,831 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:36,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:36,849 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=961): org.apache.kafka.common.requests.HeartbeatResponse@2752598 02.12.2021 17:25:36,850 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:36,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:36,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:36,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:36,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:36,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:36,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:36,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:36,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:37,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:37,038 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 200 02.12.2021 17:25:37,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:37,096 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58906ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:37,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:37,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:37,137 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239977ms for 240000ms of delay, sleeping for 33ms 02.12.2021 17:25:37,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,156 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,174 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239368ms for 240000ms of delay, sleeping for 642ms 02.12.2021 17:25:37,231 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,232 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,232 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,233 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:37,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,244 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,296 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 499 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,327 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:37,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 87 02.12.2021 17:25:37,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 398 02.12.2021 17:25:37,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:37,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:37,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:37,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:37,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:37,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,427 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:37,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:37,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:37,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:37,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:37,543 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=686): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:37,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:37,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,543 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:37,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,543 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=687) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 252 02.12.2021 17:25:37,545 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=668): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:37,550 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:37,550 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:37,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:37,550 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:37,551 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:37,551 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=669) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:37,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 276 02.12.2021 17:25:37,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:37,628 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:37,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:37,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:37,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,656 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,656 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,657 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,658 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:37,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:37,738 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,740 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:37,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:37,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,756 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,796 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:37,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:37,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:37,839 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:37,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:37,841 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239649ms for 240000ms of delay, sleeping for 361ms 02.12.2021 17:25:37,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:37,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:37,868 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:37,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:37,875 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:37,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:37,927 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:37,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:37,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:37,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:37,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:37,928 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:37,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:37,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 290 02.12.2021 17:25:37,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:37,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:38,035 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:38,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 190 02.12.2021 17:25:38,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 196 02.12.2021 17:25:38,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:38,096 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59906ms delay out of 60000ms. Remaining 94ms. Executor queue at 0 02.12.2021 17:25:38,099 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59884ms for 60000ms of delay, sleeping for 126ms 02.12.2021 17:25:38,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:38,152 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:38,159 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,160 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 165 02.12.2021 17:25:38,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:38,221 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239805ms for 240000ms of delay, sleeping for 205ms 02.12.2021 17:25:38,231 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59189ms for 60000ms of delay, sleeping for 821ms 02.12.2021 17:25:38,236 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,242 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,242 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 100 02.12.2021 17:25:38,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,257 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,258 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,258 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,259 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 65 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,297 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,297 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,332 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:38,333 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,333 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,333 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=658) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:38,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 327 02.12.2021 17:25:38,333 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@40dd5600 02.12.2021 17:25:38,333 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 327 02.12.2021 17:25:38,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 327 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,339 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 93 02.12.2021 17:25:38,342 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:38,343 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=658) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:38,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:38,345 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=658): org.apache.kafka.common.requests.HeartbeatResponse@268157f5 02.12.2021 17:25:38,345 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:38,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:38,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:38,362 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,376 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,376 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,429 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239232ms for 240000ms of delay, sleeping for 778ms 02.12.2021 17:25:38,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 226 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,435 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 454 02.12.2021 17:25:38,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:38,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:38,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 126 02.12.2021 17:25:38,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 354 02.12.2021 17:25:38,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:38,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:38,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 254 02.12.2021 17:25:38,635 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 25 02.12.2021 17:25:38,647 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:38,664 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:38,666 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:38,676 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,676 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,678 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,679 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,741 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 148 02.12.2021 17:25:38,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30324ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:38,749 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,749 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:38,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:38,768 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,768 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,769 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 116 02.12.2021 17:25:38,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:38,800 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,801 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,801 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,839 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:38,840 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:38,840 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:38,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 47 02.12.2021 17:25:38,850 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:38,869 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 16 02.12.2021 17:25:38,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:38,878 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:38,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:38,879 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:38,886 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,887 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,887 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=678) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:38,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 383 02.12.2021 17:25:38,888 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=678): org.apache.kafka.common.requests.HeartbeatResponse@5033a168 02.12.2021 17:25:38,888 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:38,888 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:38,889 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:38,889 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:38,889 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 291 02.12.2021 17:25:38,889 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,889 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=679) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:38,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 379 02.12.2021 17:25:38,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:38,896 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:38,897 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 190 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,897 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=678) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=190,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:38,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 39 02.12.2021 17:25:38,897 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=678): org.apache.kafka.common.requests.HeartbeatResponse@7756aa25 02.12.2021 17:25:38,897 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 190 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:38,898 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 86, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:38,898 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=679) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:38,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 37 02.12.2021 17:25:38,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:38,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:38,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:38,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:38,950 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:38,988 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:38,989 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:38,997 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:39,051 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 11 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,055 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,056 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147822, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:39,056 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147832, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:39,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:39,056 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:39,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,056 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,056 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 12 iterations. 02.12.2021 17:25:39,056 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59986ms for 60000ms of delay, sleeping for 24ms 02.12.2021 17:25:39,082 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59968ms for 60000ms of delay, sleeping for 42ms 02.12.2021 17:25:39,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:39,125 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59249ms for 60000ms of delay, sleeping for 761ms 02.12.2021 17:25:39,151 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:39,180 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,185 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,185 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30273ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:39,186 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 66 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,210 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 191053ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 191054ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1311): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=59, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:39,211 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:39,211 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:39,212 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975535, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:39,212 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975545, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:39,212 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1312) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:39,212 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:39,213 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:39,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,213 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,213 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 67 iterations. 02.12.2021 17:25:39,221 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239922ms for 240000ms of delay, sleeping for 88ms 02.12.2021 17:25:39,249 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,250 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,251 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,269 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,270 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,271 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,272 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,287 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,302 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,310 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239407ms for 240000ms of delay, sleeping for 603ms 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,342 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,380 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,380 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:39,489 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:39,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:39,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:39,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:39,691 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,694 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,694 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:39,754 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:39,754 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:39,754 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=962) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:39,754 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,760 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:39,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:39,803 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,804 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,804 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:39,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:39,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:39,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:39,854 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=962): org.apache.kafka.common.requests.HeartbeatResponse@49d65d0f 02.12.2021 17:25:39,870 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:39,881 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:39,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,882 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:39,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,882 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:39,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:39,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:39,883 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,896 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59833ms for 60000ms of delay, sleeping for 177ms 02.12.2021 17:25:39,898 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:39,931 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239942ms for 240000ms of delay, sleeping for 68ms 02.12.2021 17:25:39,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:39,938 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:39,938 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:39,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:39,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:39,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:39,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:39,943 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:39,944 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:39,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:40,001 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239010ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:39,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:40,069 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:40,074 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59846ms for 60000ms of delay, sleeping for 164ms 02.12.2021 17:25:40,102 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 93 02.12.2021 17:25:40,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:40,195 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,199 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,244 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59920ms for 60000ms of delay, sleeping for 90ms 02.12.2021 17:25:40,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:40,264 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,264 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,265 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:40,305 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,305 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,305 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,306 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 237 02.12.2021 17:25:40,337 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59971ms for 60000ms of delay, sleeping for 39ms 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:40,367 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:40,379 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59454ms for 60000ms of delay, sleeping for 556ms 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:40,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:40,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,451 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,467 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:40,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:40,543 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,543 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,543 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=688) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:40,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 263 02.12.2021 17:25:40,551 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:40,551 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:40,552 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=670) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:40,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 301 02.12.2021 17:25:40,568 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:40,600 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:40,668 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:40,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,700 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,701 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,701 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,702 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,707 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:40,767 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,768 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,769 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,770 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:40,779 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,786 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,790 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,797 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,808 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:40,807 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,814 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,815 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,815 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:40,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:40,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:40,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:40,890 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:40,891 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:40,908 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:40,936 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59866ms for 60000ms of delay, sleeping for 144ms 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:40,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:40,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:40,971 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:41,003 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239463ms for 240000ms of delay, sleeping for 547ms 02.12.2021 17:25:41,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:41,072 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 12 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,081 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,081 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147832, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:41,081 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147842, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:41,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:41,082 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:41,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,082 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,082 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 13 iterations. 02.12.2021 17:25:41,082 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59248ms for 60000ms of delay, sleeping for 762ms 02.12.2021 17:25:41,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:41,172 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,208 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,209 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 123 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=679): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:41,273 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:41,274 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,274 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:41,274 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,274 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=680) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1038): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:41,271 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=679): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:41,270 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,277 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:41,276 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=191, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:41,278 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:41,278 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:41,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 176 02.12.2021 17:25:41,278 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,277 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:41,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 65 02.12.2021 17:25:41,279 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1039) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,279 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=680) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:41,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 112 02.12.2021 17:25:41,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 17 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,298 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1039): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=680): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,301 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:41,301 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,301 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,302 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,302 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,302 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,302 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,302 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 150 02.12.2021 17:25:41,300 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=680): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,301 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:41,302 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:41,302 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:41,302 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,303 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=191, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:41,303 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,303 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,303 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:41,303 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:41,304 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:41,304 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,304 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,304 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,304 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,304 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,305 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:41,306 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,305 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:41,306 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:41,306 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 496 02.12.2021 17:25:41,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 23 02.12.2021 17:25:41,315 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,316 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,333 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,335 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,335 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=659) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:41,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,336 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=659): org.apache.kafka.common.requests.HeartbeatResponse@292f82b2 02.12.2021 17:25:41,336 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 374 02.12.2021 17:25:41,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 372 02.12.2021 17:25:41,344 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:41,345 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:41,345 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:41,345 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=659) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:41,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 432 02.12.2021 17:25:41,347 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=659): org.apache.kafka.common.requests.HeartbeatResponse@18e744e9 02.12.2021 17:25:41,348 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:41,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 429 02.12.2021 17:25:41,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 429 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,356 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,396 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,396 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,397 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:41,397 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:41,397 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1040) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:41,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 49 02.12.2021 17:25:41,406 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 392 02.12.2021 17:25:41,438 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 272 02.12.2021 17:25:41,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 329 02.12.2021 17:25:41,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,452 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,452 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,496 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:41,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 291 02.12.2021 17:25:41,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 172 02.12.2021 17:25:41,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 229 02.12.2021 17:25:41,555 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239415ms for 240000ms of delay, sleeping for 595ms 02.12.2021 17:25:41,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:41,570 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=669): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,570 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,570 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=687): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,571 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,570 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:41,571 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=670): org.apache.kafka.common.requests.HeartbeatResponse@2b37b8c7 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,571 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=688): org.apache.kafka.common.requests.HeartbeatResponse@2296fbf2 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:41,574 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,574 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,574 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:41,574 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 245 02.12.2021 17:25:41,574 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 242 02.12.2021 17:25:41,573 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:41,575 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:41,577 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:41,577 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:41,577 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:41,577 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:41,577 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:41,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 286 02.12.2021 17:25:41,577 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 279 02.12.2021 17:25:41,607 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:41,617 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1312): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,624 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:41,624 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=59, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:41,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 71 02.12.2021 17:25:41,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 128 02.12.2021 17:25:41,656 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:41,677 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 179 02.12.2021 17:25:41,675 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 142 02.12.2021 17:25:41,708 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 90 02.12.2021 17:25:41,713 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,717 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 27 02.12.2021 17:25:41,760 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 192 02.12.2021 17:25:41,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:41,777 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,777 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,778 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,778 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:41,786 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 30 02.12.2021 17:25:41,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:41,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:41,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,815 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,816 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,816 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,816 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,820 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,820 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,822 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:41,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,826 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:41,864 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59761ms for 60000ms of delay, sleeping for 249ms 02.12.2021 17:25:41,861 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 91 02.12.2021 17:25:41,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 513 ms and fetched 0 records 02.12.2021 17:25:41,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:41,871 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:41,871 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:41,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:41,874 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:41,875 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:41,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:41,875 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:41,879 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:41,879 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:41,879 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:41,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:41,879 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,879 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:41,886 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,886 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:41,886 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,886 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:41,886 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:41,887 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:41,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:41,899 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,916 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:41,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:41,931 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 288 02.12.2021 17:25:41,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:41,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:41,956 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:41,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:41,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:41,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:41,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:41,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:41,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:41,978 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:41,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:42,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 186 02.12.2021 17:25:42,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,076 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,079 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:42,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:42,116 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59035ms for 60000ms of delay, sleeping for 975ms 02.12.2021 17:25:42,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:42,133 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:42,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:42,151 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239432ms for 240000ms of delay, sleeping for 578ms 02.12.2021 17:25:42,176 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:42,179 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:42,187 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,220 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,226 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:42,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:42,220 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,226 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:42,227 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1313) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:42,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:42,278 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:42,281 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,281 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,282 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,282 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,322 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,326 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1313): org.apache.kafka.common.requests.HeartbeatResponse@5be252d2 02.12.2021 17:25:42,327 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,327 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,328 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,383 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:42,386 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:42,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,387 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:42,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,423 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:42,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:42,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 521 ms and fetched 0 records 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,480 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:42,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,523 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:42,527 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:42,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,581 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:42,587 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:42,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:42,623 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:42,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:42,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:42,684 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:42,687 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:42,689 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:42,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:42,720 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,735 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,735 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,739 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239840ms for 240000ms of delay, sleeping for 170ms 02.12.2021 17:25:42,779 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:42,779 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:42,779 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=963) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,787 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:42,787 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 193 02.12.2021 17:25:42,789 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:42,828 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,831 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:42,880 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=963): org.apache.kafka.common.requests.HeartbeatResponse@2f0a941a 02.12.2021 17:25:42,880 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:42,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:42,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:42,889 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:42,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:42,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:42,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,913 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238790ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:42,933 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:42,935 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:42,938 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:42,980 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:42,980 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:42,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:42,990 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:43,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:43,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,039 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:43,081 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,088 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:43,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:43,092 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59462ms for 60000ms of delay, sleeping for 548ms 02.12.2021 17:25:43,134 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:43,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:43,139 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:43,181 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,191 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:43,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:43,234 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:43,235 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,236 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,237 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,236 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:43,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,281 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:43,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,301 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,301 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 209 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,337 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:43,382 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,389 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,390 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 161 02.12.2021 17:25:43,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,401 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:43,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,434 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 109 02.12.2021 17:25:43,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:43,439 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:43,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,482 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,486 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,490 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 61 02.12.2021 17:25:43,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:43,535 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 8 02.12.2021 17:25:43,538 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:43,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:43,543 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,543 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,544 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,545 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=689) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:43,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,546 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 290 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=689): org.apache.kafka.common.requests.HeartbeatResponse@2199feb6 02.12.2021 17:25:43,552 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:43,552 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,552 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=690) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 282 02.12.2021 17:25:43,551 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:43,553 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 59 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,553 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=671) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=59,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:43,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:43,555 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=671): org.apache.kafka.common.requests.HeartbeatResponse@598950d6 02.12.2021 17:25:43,556 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:43,565 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:43,565 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,566 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,566 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 0])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,566 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=672) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:43,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 324 02.12.2021 17:25:43,589 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 397 02.12.2021 17:25:43,602 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:43,638 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:43,641 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:43,647 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,656 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,657 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59877ms for 60000ms of delay, sleeping for 133ms 02.12.2021 17:25:43,693 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 293 02.12.2021 17:25:43,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:43,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 99 02.12.2021 17:25:43,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,742 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,745 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,802 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,808 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59781ms for 60000ms of delay, sleeping for 229ms 02.12.2021 17:25:43,795 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 191 02.12.2021 17:25:43,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:43,837 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:43,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,835 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,842 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,842 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 423 02.12.2021 17:25:43,853 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,862 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:43,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:43,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:43,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:43,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,913 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:43,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:43,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,914 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,914 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:43,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 394 02.12.2021 17:25:43,913 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239790ms delay out of 240000ms. Remaining 210ms. Executor queue at 0 02.12.2021 17:25:43,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 74 02.12.2021 17:25:43,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 67 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 195770ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 195772ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:43,929 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c 02.12.2021 17:25:43,932 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:43,932 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-1], userDataSize=48) 02.12.2021 17:25:43,932 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:43,932 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Skip resuming since state is RUNNING 02.12.2021 17:25:43,932 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:43,932 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:43,932 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:43,933 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:43,933 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinPrepare with generation 59 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c 02.12.2021 17:25:43,933 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] (Re-)joining group 02.12.2021 17:25:43,934 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry3] 02.12.2021 17:25:43,934 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry3', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, -111, 35, 38, -18, 105, 96, 68, -42, -74, 55, -10, -65, 75, 79, -76, -39, 0, 0, 0, 0, 0, 0, 0, 0, 37, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1])])) to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:43,943 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1314) and timeout 305000 to node 2147483643: {group_id=staging-message-processor-retry3,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=141 cap=141]}]} 02.12.2021 17:25:43,944 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:43,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 485 02.12.2021 17:25:43,945 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Disabling heartbeat thread 02.12.2021 17:25:43,955 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 322 02.12.2021 17:25:43,964 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 281 02.12.2021 17:25:43,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:43,989 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:43,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:43,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:43,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:43,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:43,994 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:43,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:43,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 277 02.12.2021 17:25:44,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:44,039 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59027ms for 60000ms of delay, sleeping for 983ms 02.12.2021 17:25:44,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 221 02.12.2021 17:25:44,067 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 178 02.12.2021 17:25:44,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 177 02.12.2021 17:25:44,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:44,157 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 120 02.12.2021 17:25:44,169 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 76 02.12.2021 17:25:44,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 74 02.12.2021 17:25:44,216 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:44,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:44,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,247 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,247 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,248 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:44,257 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 20 02.12.2021 17:25:44,274 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,274 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,275 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=681) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:44,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 223 02.12.2021 17:25:44,275 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=681): org.apache.kafka.common.requests.HeartbeatResponse@6565c360 02.12.2021 17:25:44,275 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,276 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:44,276 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,276 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,277 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,277 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=682) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,277 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 220 02.12.2021 17:25:44,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:44,278 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 191 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=681) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=191,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:44,278 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 76 02.12.2021 17:25:44,278 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=681): org.apache.kafka.common.requests.HeartbeatResponse@5f4d7dc6 02.12.2021 17:25:44,279 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:44,279 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 191 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:44,279 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:44,279 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:44,279 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 87, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:44,279 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=682) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:44,279 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 75 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,308 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,308 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 37 02.12.2021 17:25:44,334 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,334 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,334 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=660) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:44,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 416 02.12.2021 17:25:44,334 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=660): org.apache.kafka.common.requests.HeartbeatResponse@2fb956fd 02.12.2021 17:25:44,334 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 415 02.12.2021 17:25:44,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 415 02.12.2021 17:25:44,345 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:44,346 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:44,346 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=660) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:44,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 463 02.12.2021 17:25:44,347 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=660): org.apache.kafka.common.requests.HeartbeatResponse@4048f3d 02.12.2021 17:25:44,347 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:44,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 461 02.12.2021 17:25:44,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 461 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,355 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,364 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,375 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,378 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,394 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,395 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,415 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:44,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,416 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,433 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:44,433 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:44,434 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:44,434 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:44,434 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:44,436 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:44,436 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:44,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 196280ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,435 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 314 02.12.2021 17:25:44,438 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,439 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975545, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:44,439 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975555, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:44,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 2 ms and fetched 10 records 02.12.2021 17:25:44,440 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:44,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,440 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,440 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 68 iterations. 02.12.2021 17:25:44,447 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 361 02.12.2021 17:25:44,453 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239826ms for 240000ms of delay, sleeping for 184ms 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30307ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,539 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 210 02.12.2021 17:25:44,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 260 02.12.2021 17:25:44,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 108 02.12.2021 17:25:44,643 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239645ms for 240000ms of delay, sleeping for 365ms 02.12.2021 17:25:44,649 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 159 02.12.2021 17:25:44,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 7 02.12.2021 17:25:44,750 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 58 02.12.2021 17:25:44,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,753 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,813 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:44,814 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,817 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,817 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,819 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:44,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:44,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30380ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,856 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:44,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30305ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:44,866 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:44,895 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30356ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:44,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:44,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:44,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:44,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:44,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:44,918 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:44,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:44,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:44,956 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:45,002 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30809ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,032 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59999ms for 60000ms of delay, sleeping for 11ms 02.12.2021 17:25:45,038 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239862ms for 240000ms of delay, sleeping for 148ms 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 13 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,048 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,048 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147842, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:45,049 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147852, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:45,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:45,049 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:45,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,049 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 14 iterations. 02.12.2021 17:25:45,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:45,062 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59870ms for 60000ms of delay, sleeping for 140ms 02.12.2021 17:25:45,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:45,158 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:45,196 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239755ms for 240000ms of delay, sleeping for 255ms 02.12.2021 17:25:45,205 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59330ms for 60000ms of delay, sleeping for 680ms 02.12.2021 17:25:45,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:45,254 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,255 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,323 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,323 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:45,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,357 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30882ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,358 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,358 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,366 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 30806ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,367 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,368 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 30858ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,397 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,418 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:45,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,472 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239654ms for 240000ms of delay, sleeping for 356ms 02.12.2021 17:25:45,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,504 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,504 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,505 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,506 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31311ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,506 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,506 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,506 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:45,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:45,625 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:45,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:45,693 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=672): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:45,693 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=690): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:45,701 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', members=[]) 02.12.2021 17:25:45,701 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,701 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:45,700 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', members=[]) 02.12.2021 17:25:45,705 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,705 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:45,706 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,705 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,708 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=691) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 159 02.12.2021 17:25:45,707 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=673) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 181 02.12.2021 17:25:45,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:45,772 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 517 ms and fetched 0 records 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,773 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,776 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,789 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:45,797 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:45,797 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=964) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:45,829 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239979ms for 240000ms of delay, sleeping for 31ms 02.12.2021 17:25:45,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:45,832 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,832 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,838 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,849 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 68 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 197706ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 197706ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,863 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received JOIN_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1314): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=60, protocolType=null, protocolName='stream', leader='staging-message-processor-retry3-7f1cad50-1794-4112-bb49-b2fba405b328-StreamThread-1-consumer-7edde553-5195-4ea7-8ced-95ec6caeb692', memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', members=[]) 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Enabling heartbeat thread 02.12.2021 17:25:45,864 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully joined group with generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending follower SyncGroup to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) at generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry3', generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:45,864 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975555, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:45,864 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975565, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1315) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,assignments=[]} 02.12.2021 17:25:45,864 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:45,865 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:45,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,865 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 69 iterations. 02.12.2021 17:25:45,858 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31390ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:45,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31311ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:45,872 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,866 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 398 02.12.2021 17:25:45,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,881 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239751ms for 240000ms of delay, sleeping for 259ms 02.12.2021 17:25:45,894 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59889ms for 60000ms of delay, sleeping for 121ms 02.12.2021 17:25:45,897 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=964): org.apache.kafka.common.requests.HeartbeatResponse@58eee5a5 02.12.2021 17:25:45,903 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:45,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:45,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:45,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:45,905 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,908 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:45,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31369ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:45,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:45,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:45,917 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:45,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:45,930 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:45,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:45,960 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 389 02.12.2021 17:25:45,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31813ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,019 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59888ms for 60000ms of delay, sleeping for 122ms 02.12.2021 17:25:46,063 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 289 02.12.2021 17:25:46,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:46,143 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59941ms for 60000ms of delay, sleeping for 69ms 02.12.2021 17:25:46,147 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239719ms for 240000ms of delay, sleeping for 291ms 02.12.2021 17:25:46,170 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:46,190 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:46,226 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58368ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:46,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,279 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,280 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:46,352 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,353 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,353 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,372 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 31812ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,373 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,373 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,380 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 517 ms and fetched 0 records 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,383 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31907ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,384 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,384 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 31880ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,420 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,420 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,439 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239481ms for 240000ms of delay, sleeping for 529ms 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:46,453 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,480 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32313ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,507 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,507 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:46,580 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:46,654 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:46,681 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:46,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 514 ms and fetched 0 records 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,793 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,793 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,860 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,860 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 485 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:46,874 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32313ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,883 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32409ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,885 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,885 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,884 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:46,890 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:46,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32382ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:46,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:46,921 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:46,944 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:46,945 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:46,945 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:46,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 384 02.12.2021 17:25:46,970 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239918ms for 240000ms of delay, sleeping for 92ms 02.12.2021 17:25:46,993 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32814ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,009 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 284 02.12.2021 17:25:47,063 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239016ms for 240000ms of delay, sleeping for 994ms 02.12.2021 17:25:47,094 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:47,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:47,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:47,226 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59368ms delay out of 60000ms. Remaining 632ms. Executor queue at 0 02.12.2021 17:25:47,229 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59084ms for 60000ms of delay, sleeping for 926ms 02.12.2021 17:25:47,242 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=691): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,243 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,242 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=673): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,243 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343', protocol='stream'} 02.12.2021 17:25:47,244 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 02.12.2021 17:25:47,244 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,244 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,244 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,244 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,244 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,244 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:47,244 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,243 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,245 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0', protocol='stream'} 02.12.2021 17:25:47,245 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 02.12.2021 17:25:47,245 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:47,245 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:47,245 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:47,245 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:47,245 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:47,245 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Pausing partitions [] 02.12.2021 17:25:47,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 178 02.12.2021 17:25:47,245 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 147 02.12.2021 17:25:47,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:47,270 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received SYNC_GROUP response from node 2147483643 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1315): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,271 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 51, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 48, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:47,271 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Successfully synced group in generation Generation{generationId=60, memberId='staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c', protocol='stream'} 02.12.2021 17:25:47,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,294 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,294 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,295 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 39 02.12.2021 17:25:47,335 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,339 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,339 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=661) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:47,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,340 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@540f8b00 02.12.2021 17:25:47,340 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 455 02.12.2021 17:25:47,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 455 02.12.2021 17:25:47,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:47,345 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:47,346 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:47,346 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=661) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:47,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 15 02.12.2021 17:25:47,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:47,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 43 02.12.2021 17:25:47,347 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=661): org.apache.kafka.common.requests.HeartbeatResponse@2c0bc836 02.12.2021 17:25:47,347 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:47,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 13 02.12.2021 17:25:47,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 12 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,363 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,386 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32912ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:47,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,390 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:47,391 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,392 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,392 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,394 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:47,394 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,397 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,397 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:47,398 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Committing all active tasks [] and standby tasks [] since 32838ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,422 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Resuming partitions [] 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:47,423 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:47,423 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:47,423 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Committing all active tasks [] and standby tasks [] since 32885ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,442 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 354 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:47,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:47,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:47,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:47,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33316ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,511 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,524 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:47,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 253 02.12.2021 17:25:47,565 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:47,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:47,624 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:47,642 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 153 02.12.2021 17:25:47,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:47,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:47,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:47,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 53 02.12.2021 17:25:47,775 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:47,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:47,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:47,805 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,814 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,825 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 530 ms and fetched 0 records 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33420ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:47,896 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,896 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:47,902 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:47,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:47,912 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:47,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:47,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:47,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:47,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:47,926 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,923 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:47,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:47,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:47,958 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:47,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 395 02.12.2021 17:25:48,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,011 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 33818ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,012 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:48,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:48,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 393 02.12.2021 17:25:48,061 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239905ms for 240000ms of delay, sleeping for 105ms 02.12.2021 17:25:48,099 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:48,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:48,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:48,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 293 02.12.2021 17:25:48,156 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 14 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,157 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,158 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147852, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:48,161 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147862, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:48,161 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 4 ms and fetched 10 records 02.12.2021 17:25:48,162 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:48,162 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,162 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,163 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 15 iterations. 02.12.2021 17:25:48,164 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58471ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:48,169 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239949ms for 240000ms of delay, sleeping for 61ms 02.12.2021 17:25:48,199 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 194 02.12.2021 17:25:48,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:48,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:48,231 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239952ms for 240000ms of delay, sleeping for 58ms 02.12.2021 17:25:48,231 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 69 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 200133ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 200133ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,290 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Executing onJoinComplete with generation 60 and memberId staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c 02.12.2021 17:25:48,291 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Updating assignment with 02.12.2021 17:25:48,291 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Notifying assignor about the new Assignment(partitions=[redacted.Cloud.Message.Callback.Retry3-1], userDataSize=48) 02.12.2021 17:25:48,291 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:48,291 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:330] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Skip resuming since state is RUNNING 02.12.2021 17:25:48,291 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Adding newly assigned partitions: 02.12.2021 17:25:48,291 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:48,291 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from RUNNING to REBALANCING 02.12.2021 17:25:48,291 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Pausing partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:48,291 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,291 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,300 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,314 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,314 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:48,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:48,391 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,393 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,393 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,395 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,397 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,397 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 33924ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,415 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 277 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,426 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,427 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 278 02.12.2021 17:25:48,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:48,461 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,462 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,493 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,493 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,493 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:48,497 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34319ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,513 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,513 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:48,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 176 02.12.2021 17:25:48,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 177 02.12.2021 17:25:48,593 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,595 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:48,598 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:48,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:48,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 75 02.12.2021 17:25:48,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 76 02.12.2021 17:25:48,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:48,695 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,698 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,698 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:48,704 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,710 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,704 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,716 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:48,711 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,719 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,719 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=674) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:48,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:48,710 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=692) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:48,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 204 02.12.2021 17:25:48,720 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=692): org.apache.kafka.common.requests.HeartbeatResponse@71672471 02.12.2021 17:25:48,720 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 208 02.12.2021 17:25:48,723 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=674): org.apache.kafka.common.requests.HeartbeatResponse@2d473032 02.12.2021 17:25:48,723 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:48,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 204 02.12.2021 17:25:48,724 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 203 02.12.2021 17:25:48,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 204 02.12.2021 17:25:48,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.clients.consumer.internals.Fetcher:634] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Skipping fetching records for assigned partition redacted.Cloud.Message.Callback.Retry3-1 because it is paused 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Resuming partitions [redacted.Cloud.Message.Callback.Retry3-1] 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:48,794 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:48,794 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9] State transition from REBALANCING to RUNNING 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:48,794 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committing all active tasks [0_1] and standby tasks [] since 200638ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.RecordCollectorImpl:247] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Flushing record collector 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:381] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Prepared RUNNING task for committing 02.12.2021 17:25:48,795 TRACE [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1157] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OffsetCommit request with {redacted.Cloud.Message.Callback.Retry3-1=OffsetAndMetadata{offset=975565, leaderEpoch=null, metadata='AQAAAX1787J3'}} to coordinator kafka-5.data.sta.par.eu.dc.sigfox.com:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,795 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending OFFSET_COMMIT request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1316) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null,topics=[{name=redacted.Cloud.Message.Callback.Retry3,partitions=[{partition_index=1,committed_offset=975565,committed_leader_epoch=-1,committed_metadata=AQAAAX1787J3}]}]} 02.12.2021 17:25:48,798 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:48,816 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received OFFSET_COMMIT response from node 2147483643 for request with header RequestHeader(apiKey=OFFSET_COMMIT, apiVersion=7, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1316): OffsetCommitResponseData(throttleTimeMs=0, topics=[OffsetCommitResponseTopic(name='redacted.Cloud.Message.Callback.Retry3', partitions=[OffsetCommitResponsePartition(partitionIndex=1, errorCode=0)])]) 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:1185] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Committed offset 975565 for partition redacted.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:808] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Checkpointable offsets {redacted.Cloud.Message.Callback.Retry3-1=975564} 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamTask:465] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Finalized commit for RUNNING task with eos false enforce checkpoint false 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:684] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Committed all active tasks [0_1] and standby tasks [] in 0ms 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,825 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975565, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:48,825 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975575, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:48,825 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:48,826 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:48,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,826 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,826 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 70 iterations. 02.12.2021 17:25:48,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:48,829 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239909ms for 240000ms of delay, sleeping for 101ms 02.12.2021 17:25:48,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 103 02.12.2021 17:25:48,818 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:48,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,833 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,833 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,834 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,834 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,818 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,834 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:48,834 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=965) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34425ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:48,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:48,925 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:48,925 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1317) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:48,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:48,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:48,931 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:48,932 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:48,932 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239989ms for 240000ms of delay, sleeping for 21ms 02.12.2021 17:25:48,934 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:48,942 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=965): org.apache.kafka.common.requests.HeartbeatResponse@731c42c5 02.12.2021 17:25:48,942 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:48,955 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239819ms for 240000ms of delay, sleeping for 191ms 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:48,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:48,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:48,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:48,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 34820ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,014 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,014 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,026 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1317): org.apache.kafka.common.requests.HeartbeatResponse@19a24fc 02.12.2021 17:25:49,026 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:49,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,032 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:49,034 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:49,100 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:49,132 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:49,135 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:49,148 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239052ms for 240000ms of delay, sleeping for 958ms 02.12.2021 17:25:49,165 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59472ms delay out of 60000ms. Remaining 528ms. Executor queue at 0 02.12.2021 17:25:49,170 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59064ms for 60000ms of delay, sleeping for 946ms 02.12.2021 17:25:49,203 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:49,228 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:49,233 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:49,235 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:49,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:49,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:49,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 97 02.12.2021 17:25:49,334 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,335 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,401 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,401 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 34926ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,429 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,429 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,432 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,432 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,436 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:49,463 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,463 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,463 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:49,463 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:49,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:49,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,501 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:49,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,515 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 35322ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,516 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:49,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:49,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:49,601 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:49,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:49,636 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:49,637 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:49,709 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:49,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:49,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:49,738 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:49,830 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:49,813 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,841 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,842 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 493 02.12.2021 17:25:49,844 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:49,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,903 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 35437ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:49,913 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,921 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 520 ms and fetched 0 records 02.12.2021 17:25:49,923 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,924 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,924 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 421 02.12.2021 17:25:49,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:49,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,933 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,932 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:49,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 393 02.12.2021 17:25:49,941 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:49,937 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,944 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:49,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:49,943 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:49,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:49,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:49,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:49,966 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:49,966 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:49,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:49,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:49,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:49,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:49,967 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:49,968 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 35825ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:900] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Unable to commit as we are in the middle of a rebalance, will try again when it completes. 02.12.2021 17:25:50,019 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,019 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,024 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 321 02.12.2021 17:25:50,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 292 02.12.2021 17:25:50,054 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 389 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1040): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:50,057 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,057 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1041) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=682): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:50,058 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 410 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,059 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=683) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 460 02.12.2021 17:25:50,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=682): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=192, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:50,059 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:50,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:50,060 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=683) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:50,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 358 02.12.2021 17:25:50,111 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239989ms for 240000ms of delay, sleeping for 21ms 02.12.2021 17:25:50,129 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 216 02.12.2021 17:25:50,130 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59841ms for 60000ms of delay, sleeping for 169ms 02.12.2021 17:25:50,139 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239200ms for 240000ms of delay, sleeping for 810ms 02.12.2021 17:25:50,144 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:50,155 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 288 02.12.2021 17:25:50,160 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1041): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=683): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=683): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,162 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:50,162 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,162 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,162 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,162 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,162 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,162 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,163 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 357 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:50,162 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=192, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,163 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:50,163 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:50,164 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,164 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,164 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 256 02.12.2021 17:25:50,163 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:50,164 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:50,164 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:50,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 306 02.12.2021 17:25:50,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 115 02.12.2021 17:25:50,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 91 02.12.2021 17:25:50,255 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 188 02.12.2021 17:25:50,260 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:50,263 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 256 02.12.2021 17:25:50,264 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 154 02.12.2021 17:25:50,270 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:50,300 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59299ms for 60000ms of delay, sleeping for 711ms 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,335 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,336 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,337 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,337 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,337 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=662) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:50,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 8 02.12.2021 17:25:50,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 4 02.12.2021 17:25:50,337 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=662): org.apache.kafka.common.requests.HeartbeatResponse@7ec8bad7 02.12.2021 17:25:50,337 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 4 02.12.2021 17:25:50,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 3 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,341 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,341 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,346 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,347 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,349 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:50,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:50,349 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:50,349 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=662) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:50,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 75 02.12.2021 17:25:50,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:50,350 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=662): org.apache.kafka.common.requests.HeartbeatResponse@990cbd 02.12.2021 17:25:50,350 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:50,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:50,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 74 02.12.2021 17:25:50,355 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 88 02.12.2021 17:25:50,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:50,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 156 02.12.2021 17:25:50,366 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 52 02.12.2021 17:25:50,374 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 94 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,419 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Committing all active tasks [] and standby tasks [] since 35943ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,419 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,424 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,424 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,425 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,441 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:50,445 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:50,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,459 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,460 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 55 02.12.2021 17:25:50,473 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,474 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,475 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,475 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:50,476 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:50,476 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:50,478 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:50,478 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1042) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:50,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 494 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:50,519 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:50,519 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:50,519 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:882] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Committing all active tasks [] and standby tasks [] since 36326ms has elapsed (commit interval is 30000ms) 02.12.2021 17:25:50,520 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,520 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:50,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:50,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:50,553 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 392 02.12.2021 17:25:50,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:50,574 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:50,620 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:50,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:50,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:50,648 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:50,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 292 02.12.2021 17:25:50,661 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:50,720 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:50,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:50,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:50,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:50,751 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:50,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:50,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:50,821 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:50,827 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,847 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,847 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:50,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:50,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,919 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,919 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:50,925 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:50,925 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,926 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,927 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,927 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:50,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:50,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,946 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:50,946 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:50,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:50,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:50,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,947 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 70 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 71 iterations. 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 71 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,951 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975575, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:50,951 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975585, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:50,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:50,951 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:50,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,951 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,951 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 71 iterations. 02.12.2021 17:25:50,952 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239843ms for 240000ms of delay, sleeping for 167ms 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:50,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:50,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:50,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:50,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:50,976 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,012 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59946ms for 60000ms of delay, sleeping for 64ms 02.12.2021 17:25:51,020 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,021 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,022 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,022 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,027 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:51,048 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:51,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:51,079 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59855ms for 60000ms of delay, sleeping for 155ms 02.12.2021 17:25:51,120 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239017ms for 240000ms of delay, sleeping for 993ms 02.12.2021 17:25:51,120 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:51,122 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:51,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:51,147 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:51,148 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:51,162 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:51,221 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 199 02.12.2021 17:25:51,222 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 300 02.12.2021 17:25:51,227 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 200 02.12.2021 17:25:51,236 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59373ms for 60000ms of delay, sleeping for 637ms 02.12.2021 17:25:51,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:51,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 99 02.12.2021 17:25:51,262 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:51,322 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:51,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:51,333 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:51,347 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,348 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,348 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:51,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:51,420 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,420 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,420 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,424 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,428 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,428 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,447 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,448 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,449 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,449 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 255 02.12.2021 17:25:51,448 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,462 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,463 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,468 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,468 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,469 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,469 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 242 02.12.2021 17:25:51,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:51,477 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,478 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,478 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,522 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:51,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:51,524 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,525 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,525 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,529 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:51,549 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 155 02.12.2021 17:25:51,552 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:51,569 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 142 02.12.2021 17:25:51,622 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:51,626 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:51,629 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:51,652 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 52 02.12.2021 17:25:51,653 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:51,680 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 31 02.12.2021 17:25:51,705 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,716 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,717 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=693) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:51,717 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 244 02.12.2021 17:25:51,718 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=693): org.apache.kafka.common.requests.HeartbeatResponse@5153da6a 02.12.2021 17:25:51,722 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 227 02.12.2021 17:25:51,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 227 02.12.2021 17:25:51,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:51,719 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,733 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,734 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=675) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:51,719 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:51,734 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=675): org.apache.kafka.common.requests.HeartbeatResponse@79bbf4da 02.12.2021 17:25:51,734 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:51,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 235 02.12.2021 17:25:51,730 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:51,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 297 02.12.2021 17:25:51,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 232 02.12.2021 17:25:51,753 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:51,822 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 127 02.12.2021 17:25:51,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 187 02.12.2021 17:25:51,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:51,826 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:51,851 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:51,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,853 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,853 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,849 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,857 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:51,857 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=966) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:51,854 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 115 02.12.2021 17:25:51,854 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,863 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,874 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59825ms for 60000ms of delay, sleeping for 185ms 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,929 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:51,930 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 19 02.12.2021 17:25:51,932 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:51,933 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:51,933 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1318) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:51,939 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 86 02.12.2021 17:25:51,940 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 519 ms and fetched 0 records 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,942 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,942 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:51,950 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:51,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:51,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:51,963 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 6 02.12.2021 17:25:51,961 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=966): org.apache.kafka.common.requests.HeartbeatResponse@859578c 02.12.2021 17:25:51,965 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:51,970 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:51,970 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:51,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:51,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:51,982 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:51,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:51,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:51,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:51,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:51,983 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:51,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,026 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,027 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,028 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,028 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,030 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:52,033 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1318): org.apache.kafka.common.requests.HeartbeatResponse@45627b9f 02.12.2021 17:25:52,034 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:52,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:52,057 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 15 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:52,060 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147862, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:52,060 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147872, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:52,060 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:52,061 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:52,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,061 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:52,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,061 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 16 iterations. 02.12.2021 17:25:52,062 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59951ms for 60000ms of delay, sleeping for 59ms 02.12.2021 17:25:52,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,070 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,115 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 238228ms for 240000ms of delay, sleeping for 1000ms 02.12.2021 17:25:52,122 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59874ms for 60000ms of delay, sleeping for 136ms 02.12.2021 17:25:52,129 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:52,130 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,143 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,158 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:52,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:52,171 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:52,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,230 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:52,244 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:52,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:52,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 98 02.12.2021 17:25:52,271 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59574ms for 60000ms of delay, sleeping for 436ms 02.12.2021 17:25:52,271 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,330 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:52,334 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:52,345 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 97 02.12.2021 17:25:52,364 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 93 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,365 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,365 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,378 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,430 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:52,430 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:52,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:52,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,444 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,444 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,445 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,446 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,458 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,471 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,483 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:52,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,529 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,529 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,530 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:52,530 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,532 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:52,547 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 399 02.12.2021 17:25:52,558 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:52,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:52,572 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:52,630 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:52,633 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:52,647 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,658 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:52,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:52,673 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:52,717 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59785ms for 60000ms of delay, sleeping for 225ms 02.12.2021 17:25:52,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:52,733 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 299 02.12.2021 17:25:52,748 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 198 02.12.2021 17:25:52,759 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:52,771 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:52,774 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:52,833 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:52,838 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 194 02.12.2021 17:25:52,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 98 02.12.2021 17:25:52,865 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,869 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,869 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,871 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:52,871 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,872 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,873 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 464 02.12.2021 17:25:52,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:52,935 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,936 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,936 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 413 02.12.2021 17:25:52,940 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 92 02.12.2021 17:25:52,943 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59908ms for 60000ms of delay, sleeping for 102ms 02.12.2021 17:25:52,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:52,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,949 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,949 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 110 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:52,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:52,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,972 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:52,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:52,973 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:52,974 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 363 02.12.2021 17:25:52,984 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:52,985 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:52,985 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,033 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,033 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 26 02.12.2021 17:25:53,036 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 313 02.12.2021 17:25:53,046 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59334ms for 60000ms of delay, sleeping for 676ms 02.12.2021 17:25:53,049 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 10 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 0 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,059 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=684) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null} 02.12.2021 17:25:53,059 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry2] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending Heartbeat request with generation 192 and member id staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,060 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 474 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=684) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=192,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null} 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=684): org.apache.kafka.common.requests.HeartbeatResponse@22a3d0d 02.12.2021 17:25:53,060 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:53,060 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=685) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 390 02.12.2021 17:25:53,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 472 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received HEARTBEAT response from node 2147483645 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=684): org.apache.kafka.common.requests.HeartbeatResponse@6598d53e 02.12.2021 17:25:53,061 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1110] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Attempt to heartbeat failed since group is rebalancing 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 192 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:53,061 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 88, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:53,061 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=685) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:53,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 388 02.12.2021 17:25:53,073 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,074 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 263 02.12.2021 17:25:53,116 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 239229ms delay out of 240000ms. Remaining 771ms. Executor queue at 0 02.12.2021 17:25:53,117 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239132ms for 240000ms of delay, sleeping for 878ms 02.12.2021 17:25:53,136 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 213 02.12.2021 17:25:53,159 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:53,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,160 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:53,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,174 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 163 02.12.2021 17:25:53,237 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 112 02.12.2021 17:25:53,259 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 200 02.12.2021 17:25:53,274 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:53,275 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 62 02.12.2021 17:25:53,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 12 02.12.2021 17:25:53,337 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,337 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,338 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,338 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=663) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:53,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 35 02.12.2021 17:25:53,339 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@2b278dae 02.12.2021 17:25:53,339 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 34 02.12.2021 17:25:53,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:53,349 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:53,350 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:53,350 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=663) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:53,349 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:53,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:53,353 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=663): org.apache.kafka.common.requests.HeartbeatResponse@22fbd04b 02.12.2021 17:25:53,353 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:53,356 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 83 02.12.2021 17:25:53,357 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 79 02.12.2021 17:25:53,360 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:53,375 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:53,374 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,375 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,375 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,436 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,437 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,437 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,450 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,451 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,460 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,475 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,475 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,477 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,485 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:53,485 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,533 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,534 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,534 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,537 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:53,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:53,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:53,578 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:53,639 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:53,661 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:53,679 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:53,678 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:53,725 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59709ms for 60000ms of delay, sleeping for 301ms 02.12.2021 17:25:53,739 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:53,761 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:53,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 94 02.12.2021 17:25:53,781 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 194 02.12.2021 17:25:53,851 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:53,862 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:53,875 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,877 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,878 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,886 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,894 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 81 02.12.2021 17:25:53,947 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 510 ms and fetched 0 records 02.12.2021 17:25:53,948 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,953 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,948 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,953 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,954 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,954 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:53,960 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:53,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,963 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:53,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:53,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:53,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:53,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,976 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,979 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:53,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:53,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:53,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:53,987 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:53,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:53,986 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:53,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:53,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:53,998 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239792ms for 240000ms of delay, sleeping for 218ms 02.12.2021 17:25:54,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 16 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:54,030 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,031 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,031 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147872, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:54,031 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:54,032 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:54,034 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,036 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:54,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,036 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 17 iterations. 02.12.2021 17:25:54,036 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,042 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,045 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,045 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59996ms for 60000ms of delay, sleeping for 14ms 02.12.2021 17:25:54,046 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,047 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:54,061 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59055ms for 60000ms of delay, sleeping for 955ms 02.12.2021 17:25:54,066 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:54,087 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,090 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 393 02.12.2021 17:25:54,156 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 298 02.12.2021 17:25:54,168 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:54,194 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 289 02.12.2021 17:25:54,189 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:54,222 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239328ms for 240000ms of delay, sleeping for 682ms 02.12.2021 17:25:54,256 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:54,268 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 197 02.12.2021 17:25:54,298 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 185 02.12.2021 17:25:54,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 87 02.12.2021 17:25:54,359 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:54,371 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:54,386 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,387 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,388 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,398 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 85 02.12.2021 17:25:54,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,455 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,455 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,456 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,454 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,456 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,457 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,458 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,470 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:54,471 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,472 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,472 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 233 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,484 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,484 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 235 02.12.2021 17:25:54,488 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,493 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,493 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:54,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,494 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,548 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,556 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:54,573 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 132 02.12.2021 17:25:54,588 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 300 02.12.2021 17:25:54,586 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 134 02.12.2021 17:25:54,657 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:54,683 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 22 02.12.2021 17:25:54,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:54,690 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 29 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,712 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,713 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,713 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,713 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=694) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:54,714 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 259 02.12.2021 17:25:54,727 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=694): org.apache.kafka.common.requests.HeartbeatResponse@448fe2d9 02.12.2021 17:25:54,727 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 245 02.12.2021 17:25:54,727 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 245 02.12.2021 17:25:54,725 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,731 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,725 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:54,731 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=676) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:54,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 250 02.12.2021 17:25:54,734 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=676): org.apache.kafka.common.requests.HeartbeatResponse@6f5c9aaa 02.12.2021 17:25:54,734 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:54,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 250 02.12.2021 17:25:54,734 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 250 02.12.2021 17:25:54,763 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:54,791 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:54,828 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 144 02.12.2021 17:25:54,835 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 149 02.12.2021 17:25:54,864 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:54,871 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,881 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:54,881 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=967) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:54,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:54,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,898 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,898 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,899 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,899 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,923 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239593ms for 240000ms of delay, sleeping for 417ms 02.12.2021 17:25:54,929 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 43 02.12.2021 17:25:54,941 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:54,941 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:54,942 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1319) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:54,948 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 36 02.12.2021 17:25:54,956 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,957 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,957 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:54,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,958 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,958 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,959 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:54,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 519 ms and fetched 0 records 02.12.2021 17:25:54,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,991 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:54,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:54,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:54,992 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:54,992 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,991 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=967): org.apache.kafka.common.requests.HeartbeatResponse@7308b8d9 02.12.2021 17:25:54,991 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:54,997 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:54,997 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:54,995 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:54,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:54,999 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,039 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59093ms for 60000ms of delay, sleeping for 917ms 02.12.2021 17:25:55,045 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1319): org.apache.kafka.common.requests.HeartbeatResponse@6b55c528 02.12.2021 17:25:55,051 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:55,061 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 396 02.12.2021 17:25:55,049 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,064 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,064 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,070 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,071 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,097 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:55,161 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 296 02.12.2021 17:25:55,193 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:55,197 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:55,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 195 02.12.2021 17:25:55,261 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:55,295 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:55,299 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:55,304 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:55,343 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239200ms for 240000ms of delay, sleeping for 810ms 02.12.2021 17:25:55,363 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:55,396 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 96 02.12.2021 17:25:55,399 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:55,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,402 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,459 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,459 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,460 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,460 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,494 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,494 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,495 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,495 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,502 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,503 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:55,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,503 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,504 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 537 ms and fetched 0 records 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:55,536 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:55,536 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,560 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,576 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,576 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,595 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:55,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:55,662 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 297 02.12.2021 17:25:55,696 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:55,706 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:55,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:55,762 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 197 02.12.2021 17:25:55,796 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:55,806 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:55,807 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:55,878 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 81 02.12.2021 17:25:55,903 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,904 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,904 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 91 02.12.2021 17:25:55,904 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,906 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,906 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:55,907 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,912 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 425 02.12.2021 17:25:55,959 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,962 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 387 02.12.2021 17:25:55,961 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:55,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:55,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:55,977 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59974ms for 60000ms of delay, sleeping for 36ms 02.12.2021 17:25:55,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:55,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:55,998 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,012 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 325 02.12.2021 17:25:56,022 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59758ms for 60000ms of delay, sleeping for 252ms 02.12.2021 17:25:56,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:56,038 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,040 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,040 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,064 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 285 02.12.2021 17:25:56,085 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,089 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,089 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,101 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 397 02.12.2021 17:25:56,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:56,115 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 222 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 71 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 72 iterations. 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 72 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,154 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,154 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975585, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:56,154 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975595, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:56,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 1 ms and fetched 10 records 02.12.2021 17:25:56,155 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:56,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,155 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,155 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 72 iterations. 02.12.2021 17:25:56,157 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239977ms for 240000ms of delay, sleeping for 33ms 02.12.2021 17:25:56,165 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 184 02.12.2021 17:25:56,193 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239990ms for 240000ms of delay, sleeping for 20ms 02.12.2021 17:25:56,202 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:56,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:56,215 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239989ms for 240000ms of delay, sleeping for 21ms 02.12.2021 17:25:56,215 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 122 02.12.2021 17:25:56,239 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239463ms for 240000ms of delay, sleeping for 547ms 02.12.2021 17:25:56,265 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 84 02.12.2021 17:25:56,287 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59567ms for 60000ms of delay, sleeping for 443ms 02.12.2021 17:25:56,303 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 195 02.12.2021 17:25:56,311 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 196 02.12.2021 17:25:56,315 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 22 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,338 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,339 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,340 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,340 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=664) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:56,340 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@4ecc3889 02.12.2021 17:25:56,340 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:56,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 72 02.12.2021 17:25:56,350 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:56,350 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:56,350 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=664) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:56,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:56,351 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=664): org.apache.kafka.common.requests.HeartbeatResponse@2a6ce4e 02.12.2021 17:25:56,351 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:56,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 111 02.12.2021 17:25:56,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 110 02.12.2021 17:25:56,404 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 94 02.12.2021 17:25:56,412 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 95 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,413 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,414 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,414 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,452 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 10 02.12.2021 17:25:56,463 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,464 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,464 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,466 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,474 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,474 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,499 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,499 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,508 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:56,515 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,541 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,542 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,542 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,564 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:56,589 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,590 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,591 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,599 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,608 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:56,615 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:56,665 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:56,699 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:56,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 297 02.12.2021 17:25:56,715 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 199 02.12.2021 17:25:56,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 10 records with 17 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,761 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:56,763 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processed 0 records with 18 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,763 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:56,763 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:56,773 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:56,781 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Returning 10 fetched records at offset FetchPosition{offset=1147882, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:56,788 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Update fetching position to FetchPosition{offset=1147892, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-4.data.sta.redacted:9093 (id: 3 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry1-5 02.12.2021 17:25:56,789 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239744ms for 240000ms of delay, sleeping for 266ms 02.12.2021 17:25:56,788 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Main Consumer poll completed in 25 ms and fetched 10 records 02.12.2021 17:25:56,800 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 199 02.12.2021 17:25:56,806 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] task [0_5] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry1-5, new queue size is 10 02.12.2021 17:25:56,806 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,806 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:56,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,816 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1] Processing tasks with 18 iterations. 02.12.2021 17:25:56,817 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 191 02.12.2021 17:25:56,818 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 96 02.12.2021 17:25:56,819 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58820ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:56,882 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 82 02.12.2021 17:25:56,900 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 99 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,915 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,915 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,918 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 90 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:56,965 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:56,965 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:56,975 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:56,975 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:56,999 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,000 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,000 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,008 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:57,016 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:57,042 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,043 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:57,043 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,065 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239160ms for 240000ms of delay, sleeping for 850ms 02.12.2021 17:25:57,065 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:57,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,091 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,091 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,092 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,092 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,100 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:57,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:57,116 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,166 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,200 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:57,209 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:57,217 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:57,269 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 196 02.12.2021 17:25:57,302 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:57,310 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:57,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:57,370 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 95 02.12.2021 17:25:57,402 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:57,410 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,415 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,416 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,465 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,465 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,476 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,476 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,500 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,501 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,501 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,502 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,502 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 211 02.12.2021 17:25:57,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,510 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,511 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 214 02.12.2021 17:25:57,516 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,543 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,549 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:57,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:57,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:57,550 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:57,551 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,566 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 399 02.12.2021 17:25:57,592 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:57,603 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 110 02.12.2021 17:25:57,601 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,604 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,605 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,605 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,605 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,606 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,611 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 114 02.12.2021 17:25:57,616 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,666 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:57,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 10 02.12.2021 17:25:57,713 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,713 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,713 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 13 02.12.2021 17:25:57,722 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 193 02.12.2021 17:25:57,715 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=695) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer-92ebdb4c-95e2-415a-b92d-29016b613343,group_instance_id=null} 02.12.2021 17:25:57,728 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,730 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0 to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,728 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 0 02.12.2021 17:25:57,726 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 276 02.12.2021 17:25:57,730 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, correlationId=695): org.apache.kafka.common.requests.HeartbeatResponse@2a0dc8b0 02.12.2021 17:25:57,730 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,730 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=677) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer-440ec97f-7a23-45d3-9d38-b4da4dcf85c0,group_instance_id=null} 02.12.2021 17:25:57,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 272 02.12.2021 17:25:57,743 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 259 02.12.2021 17:25:57,742 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 269 02.12.2021 17:25:57,747 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, correlationId=677): org.apache.kafka.common.requests.HeartbeatResponse@7cc06877 02.12.2021 17:25:57,747 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:57,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 264 02.12.2021 17:25:57,747 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 264 02.12.2021 17:25:57,777 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:57,820 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:51] Sleep was not enough, scheduling execution for message with 59821ms delay out of 60000ms. Remaining 179ms. Executor queue at 0 02.12.2021 17:25:57,821 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59705ms for 60000ms of delay, sleeping for 305ms 02.12.2021 17:25:57,823 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 92 02.12.2021 17:25:57,848 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 163 02.12.2021 17:25:57,846 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 156 02.12.2021 17:25:57,877 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:57,911 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,912 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:57,912 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=968) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer-bf85534f-602a-429e-b5b7-0e53c060e7ae,group_instance_id=null} 02.12.2021 17:25:57,916 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,917 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,917 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,920 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,920 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:57,919 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239875ms for 240000ms of delay, sleeping for 135ms 02.12.2021 17:25:57,952 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 51 02.12.2021 17:25:57,951 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 60 02.12.2021 17:25:57,957 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry3] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:57,957 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending Heartbeat request with generation 60 and member id staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c to coordinator kafka-5.data.sta.redacted:9093 (id: 2147483643 rack: null) 02.12.2021 17:25:57,957 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1320) and timeout 30000 to node 2147483643: {group_id=staging-message-processor-retry3,generation_id=60,member_id=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer-670b4277-39ec-45bc-8456-1ca3a2d0e51c,group_instance_id=null} 02.12.2021 17:25:57,973 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 508 ms and fetched 0 records 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:57,979 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:57,979 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:57,977 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:57,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:57,981 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:57,982 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 506 ms and fetched 0 records 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,008 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,009 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,012 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, correlationId=968): org.apache.kafka.common.requests.HeartbeatResponse@5c64a8d0 02.12.2021 17:25:58,015 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-1-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:58,017 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 507 ms and fetched 0 records 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,018 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,018 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,053 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,056 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 499 02.12.2021 17:25:58,058 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received HEARTBEAT response from node 2147483643 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, correlationId=1320): org.apache.kafka.common.requests.HeartbeatResponse@22e87e52 02.12.2021 17:25:58,060 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Received successful Heartbeat response 02.12.2021 17:25:58,066 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239114ms for 240000ms of delay, sleeping for 896ms 02.12.2021 17:25:58,082 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:58,109 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,111 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,112 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,112 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,118 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 400 02.12.2021 17:25:58,127 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:58,129 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59756ms for 60000ms of delay, sleeping for 254ms 02.12.2021 17:25:58,183 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 299 02.12.2021 17:25:58,210 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 299 02.12.2021 17:25:58,219 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 300 02.12.2021 17:25:58,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 191 02.12.2021 17:25:58,284 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 198 02.12.2021 17:25:58,317 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 192 02.12.2021 17:25:58,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 187 02.12.2021 17:25:58,331 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 90 02.12.2021 17:25:58,384 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59758ms for 60000ms of delay, sleeping for 252ms 02.12.2021 17:25:58,385 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 97 02.12.2021 17:25:58,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 92 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,421 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,421 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:58,431 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 87 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,481 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,481 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,564 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 556 ms and fetched 0 records 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,736 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,736 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,563 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 582 ms and fetched 0 records 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,557 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,737 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 372 02.12.2021 17:25:58,549 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 531 ms and fetched 0 records 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,738 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,824 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:58,824 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:58,737 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,654 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 542 ms and fetched 0 records 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:58,897 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:58,897 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:58,888 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 462 02.12.2021 17:25:58,841 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 80 02.12.2021 17:25:58,837 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:58,920 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 59151ms for 60000ms of delay, sleeping for 859ms 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 10 records with 72 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 73 iterations. 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processed 0 records with 73 iterations; invoking punctuators if necessary 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:58,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:689] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Returning 10 fetched records at offset FetchPosition{offset=975595, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for assigned partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:58,963 TRACE [org.apache.kafka.clients.consumer.internals.Fetcher:697] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1-consumer, groupId=staging-message-processor-retry3] Update fetching position to FetchPosition{offset=975605, offsetEpoch=Optional[0], currentLeader=LeaderAndEpoch{leader=Optional[kafka-3.data.sta.redacted:9093 (id: 2 rack: null)], epoch=absent}} for partition OSS.STAGING.Cloud.Message.Callback.Retry3-1 02.12.2021 17:25:58,963 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Main Consumer poll completed in 0 ms and fetched 10 records 02.12.2021 17:25:58,964 TRACE [org.apache.kafka.streams.processor.internals.StreamTask:891] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] task [0_1] Added records into the buffered queue of partition redacted.Cloud.Message.Callback.Retry3-1, new queue size is 10 02.12.2021 17:25:58,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:58,964 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:58,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:58,964 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-1] Processing tasks with 73 iterations. 02.12.2021 17:25:58,964 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239016ms for 240000ms of delay, sleeping for 994ms 02.12.2021 17:25:58,983 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 341 02.12.2021 17:25:58,982 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:58,998 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,003 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 347 02.12.2021 17:25:58,998 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:58,997 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 576 ms and fetched 0 records 02.12.2021 17:25:59,005 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,007 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 229 02.12.2021 17:25:59,003 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,007 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,007 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,011 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,021 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 319 02.12.2021 17:25:59,083 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 241 02.12.2021 17:25:59,103 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 247 02.12.2021 17:25:59,107 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 129 02.12.2021 17:25:59,128 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 212 02.12.2021 17:25:59,184 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 140 02.12.2021 17:25:59,208 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 28 02.12.2021 17:25:59,204 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 146 02.12.2021 17:25:59,229 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 111 02.12.2021 17:25:59,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,238 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:59,238 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:59,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:59,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,245 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,246 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:59,249 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,248 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,286 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 38 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1042): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=685): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:59,283 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received JOIN_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=685): JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:59,286 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', members=[]) 02.12.2021 17:25:59,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,286 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', members=[]) 02.12.2021 17:25:59,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,287 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:59,286 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:576] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful JoinGroup response: JoinGroupResponseData(throttleTimeMs=0, errorCode=0, generationId=193, protocolType=null, protocolName='stream', leader='staging-message-processor-retry2-f055f10f-2377-42c8-a04b-a6080c326d2b-StreamThread-2-consumer-2af19565-f553-4935-a843-ff3b226b375c', memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', members=[]) 02.12.2021 17:25:59,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1303] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Enabling heartbeat thread 02.12.2021 17:25:59,287 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:59,287 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,287 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:596] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully joined group with generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:59,289 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=686) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,289 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:682] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending follower SyncGroup to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) at generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'}: SyncGroupRequestData(groupId='staging-message-processor-retry2', generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocolName='stream', assignments=[]) 02.12.2021 17:25:59,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1043) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,290 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Sending SYNC_GROUP request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=686) and timeout 30000 to node 2147483645: {group_id=staging-message-processor-retry2,generation_id=193,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da,group_instance_id=null,assignments=[]} 02.12.2021 17:25:59,289 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 108 02.12.2021 17:25:59,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 458 02.12.2021 17:25:59,292 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 215 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, correlationId=686): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,315 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23', protocol='stream'} 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer-b407648f-572e-42e0-8180-3891b5f1dc23 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,316 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,316 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,316 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,316 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from RUNNING to REBALANCING 02.12.2021 17:25:59,316 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,316 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 82 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1043): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:59,316 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 02.12.2021 17:25:59,315 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received SYNC_GROUP response from node 2147483645 for request with header RequestHeader(apiKey=SYNC_GROUP, apiVersion=3, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, correlationId=686): SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', protocol='stream'} 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:740] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Received successful SyncGroup response: SyncGroupResponseData(throttleTimeMs=0, errorCode=0, protocolType=null, protocolName=null, assignment=[0, 1, 0, 0, 0, 0, 0, 0, 0, 40, 0, 0, 0, 8, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 127, -1, -1, -1, -1, -1, -1, -1]) 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1474] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer] Requested to schedule immediate rebalance for new tasks to be safely revoked from current owner. 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Successfully synced group in generation Generation{generationId=193, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da', protocol='stream'} 02.12.2021 17:25:59,317 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,317 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 33 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,317 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:349] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Executing onJoinComplete with generation 193 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer-ca6ce0c2-b63f-4be2-8e08-3757ca2aa9da 02.12.2021 17:25:59,318 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 433 02.12.2021 17:25:59,318 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:393] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Updating assignment with 02.12.2021 17:25:59,318 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:276] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Notifying assignor about the new Assignment(partitions=[], userDataSize=40) 02.12.2021 17:25:59,318 INFO [org.apache.kafka.streams.processor.internals.StreamsPartitionAssignor:1480] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer] No followup rebalance was requested, resetting the rebalance schedule. 02.12.2021 17:25:59,318 INFO [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:288] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Adding newly assigned partitions: 02.12.2021 17:25:59,318 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from RUNNING to PARTITIONS_ASSIGNED 02.12.2021 17:25:59,319 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Pausing partitions [] 02.12.2021 17:25:59,319 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 190 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 500 ms and fetched 0 records 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,324 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,324 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,329 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 11 02.12.2021 17:25:59,340 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353 to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,342 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=665) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer-dd7b1613-1396-4958-afa9-da1aa5349353,group_instance_id=null} 02.12.2021 17:25:59,340 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:59,342 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, correlationId=665): org.apache.kafka.common.requests.HeartbeatResponse@66b3fddb 02.12.2021 17:25:59,342 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,342 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 179 02.12.2021 17:25:59,343 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 178 02.12.2021 17:25:59,350 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:59,351 TRACE [org.apache.kafka.clients.consumer.internals.Heartbeat:78] [Heartbeat groupID=staging-message-processor-retry1] Sending heartbeat request with 3000ms remaining on timer 02.12.2021 17:25:59,351 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1075] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending Heartbeat request with generation 43 and member id staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a to coordinator kafka-4.data.sta.redacted:9093 (id: 2147483644 rack: null) 02.12.2021 17:25:59,351 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Sending HEARTBEAT request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=665) and timeout 30000 to node 2147483644: {group_id=staging-message-processor-retry1,generation_id=43,member_id=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer-653bb838-f0d1-40b9-9741-8ad9a1b9725a,group_instance_id=null} 02.12.2021 17:25:59,351 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 0 02.12.2021 17:25:59,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 36 02.12.2021 17:25:59,352 DEBUG [org.apache.kafka.clients.NetworkClient:884] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received HEARTBEAT response from node 2147483644 for request with header RequestHeader(apiKey=HEARTBEAT, apiVersion=3, clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, correlationId=665): org.apache.kafka.common.requests.HeartbeatResponse@709efc54 02.12.2021 17:25:59,352 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1098] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Received successful Heartbeat response 02.12.2021 17:25:59,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 36 02.12.2021 17:25:59,352 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 36 02.12.2021 17:25:59,388 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 651 ms and fetched 0 records 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,390 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,391 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 502 ms and fetched 0 records 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to restore active tasks: {} 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Transiting to update standby tasks: {} 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,399 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Initialization call done. State is RUNNING 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,399 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,400 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,400 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,418 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 331 02.12.2021 17:25:59,419 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 88 02.12.2021 17:25:59,425 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 399 02.12.2021 17:25:59,443 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 78 02.12.2021 17:25:59,450 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 298 02.12.2021 17:25:59,491 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400 02.12.2021 17:25:59,500 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:59,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Main Consumer poll completed in 505 ms and fetched 0 records 02.12.2021 17:25:59,508 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to restore active tasks: {} 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Transiting to update standby tasks: {} 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,509 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Initialization call done. State is RUNNING 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,509 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,509 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,518 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 231 02.12.2021 17:25:59,522 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Main Consumer poll completed in 511 ms and fetched 0 records 02.12.2021 17:25:59,523 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,524 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,528 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 296 02.12.2021 17:25:59,531 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,535 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,538 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,550 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 198 02.12.2021 17:25:59,548 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,597 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 294 02.12.2021 17:25:59,604 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 296 02.12.2021 17:25:59,610 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 400 02.12.2021 17:25:59,621 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 128 02.12.2021 17:25:59,631 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 193 02.12.2021 17:25:59,651 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 397 02.12.2021 17:25:59,650 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 98 02.12.2021 17:25:59,703 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 188 02.12.2021 17:25:59,705 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 195 02.12.2021 17:25:59,711 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 298 02.12.2021 17:25:59,723 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 26 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Main Consumer poll completed in 504 ms and fetched 0 records 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:723] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State is PARTITIONS_ASSIGNED; initializing tasks if necessary 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:295] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to restore active tasks: {} 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2017] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Pausing partitions [] 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:765] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Paused partitions [] from the restore consumer 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Resuming partitions [] 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:321] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Transiting to update standby tasks: {} 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.clients.consumer.KafkaConsumer:2037] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-restore-consumer, groupId=null] Resuming partitions [] 02.12.2021 17:25:59,750 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:790] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Resumed partitions [] from the restore consumer 02.12.2021 17:25:59,750 INFO [org.apache.kafka.streams.processor.internals.StreamThread:221] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING 02.12.2021 17:25:59,750 INFO [org.apache.kafka.streams.KafkaStreams:281] stream-client [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21] State transition from REBALANCING to RUNNING 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:735] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Initialization call done. State is RUNNING 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Finished restoring all changelogs [] 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processing tasks with 56 iterations. 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Processed 0 records with 56 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] 0 punctuators ran. 02.12.2021 17:25:59,751 INFO [org.apache.kafka.streams.processor.internals.StreamThread:555] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Triggering the followup rebalance scheduled for 0 ms. 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1] Invoking poll on main Consumer 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:674] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Executing onJoinPrepare with generation 193 and memberId staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9 02.12.2021 17:25:59,751 INFO [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] (Re-)joining group 02.12.2021 17:25:59,755 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 293 02.12.2021 17:25:59,755 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Main Consumer poll completed in 509 ms and fetched 0 records 02.12.2021 17:25:59,756 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,754 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 70 02.12.2021 17:25:59,751 DEBUG [org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:215] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Joining group with current subscription: [redacted.Cloud.Message.Callback.Retry2] 02.12.2021 17:25:59,757 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:552] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JoinGroup (JoinGroupRequestData(groupId='staging-message-processor-retry2', sessionTimeoutMs=20000, rebalanceTimeoutMs=300000, memberId='staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9', groupInstanceId=null, protocolType='consumer', protocols=[JoinGroupRequestProtocol(name='stream', metadata=[0, 1, 0, 0, 0, 1, 0, 41, 79, 83, 83, 46, 83, 84, 65, 71, 73, 78, 71, 46, 67, 108, 111, 117, 100, 46, 77, 101, 115, 115, 97, 103, 101, 46, 67, 97, 108, 108, 98, 97, 99, 107, 46, 82, 101, 116, 114, 121, 50, 0, 0, 0, 33, 0, 0, 0, 8, 0, 0, 0, 8, 45, -114, 121, -36, -9, -120, 66, -122, -100, 100, -20, 98, 16, -95, -1, 33, 0, 0, 0, 0, 0, 0, 0, 0, 89, 0, 0, 0, 0])])) to coordinator kafka-3.data.sta.redacted:9093 (id: 2147483645 rack: null) 02.12.2021 17:25:59,757 DEBUG [org.apache.kafka.clients.NetworkClient:522] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Sending JOIN_GROUP request with header RequestHeader(apiKey=JOIN_GROUP, apiVersion=5, clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, correlationId=1044) and timeout 305000 to node 2147483645: {group_id=staging-message-processor-retry2,session_timeout_ms=20000,rebalance_timeout_ms=300000,member_id=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer-c3598991-7f76-4c24-82e1-a7f442fd14f9,group_instance_id=null,protocol_type=consumer,protocols=[{name=stream,metadata=java.nio.HeapByteBuffer[pos=0 lim=90 cap=90]}]} 02.12.2021 17:25:59,757 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 494 02.12.2021 17:25:59,756 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,771 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,774 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,782 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,793 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 58736ms for 60000ms of delay, sleeping for 1000ms 02.12.2021 17:25:59,803 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 88 02.12.2021 17:25:59,805 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 95 02.12.2021 17:25:59,812 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 197 02.12.2021 17:25:59,828 DEBUG [org.apache.kafka.clients.consumer.internals.AbstractCoordinator:1312] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-1-consumer, groupId=staging-message-processor-retry2] Disabling heartbeat thread 02.12.2021 17:25:59,827 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:59,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,844 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Finished restoring all changelogs [] 02.12.2021 17:25:59,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,844 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processing tasks with 1 iterations. 02.12.2021 17:25:59,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] 0 punctuators ran. 02.12.2021 17:25:59,855 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3] Invoking poll on main Consumer 02.12.2021 17:25:59,857 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 500 02.12.2021 17:25:59,856 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 192 02.12.2021 17:25:59,884 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 398 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Main Consumer poll completed in 503 ms and fetched 0 records 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,893 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,893 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 500 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:777] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Main Consumer poll completed in 501 ms and fetched 0 records 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:740] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotently invoking restoration logic in state RUNNING 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StoreChangelogReader:421] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Finished restoring all changelogs [] 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:745] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Idempotent restore call done. Thread state has not changed. 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:646] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processing tasks with 1 iterations. 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:664] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Processed 0 records with 1 iterations; invoking punctuators if necessary 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:675] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] 0 punctuators ran. 02.12.2021 17:25:59,901 DEBUG [org.apache.kafka.streams.processor.internals.StreamThread:750] stream-thread [staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2] Invoking poll on main Consumer 02.12.2021 17:25:59,901 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-2-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 500 02.12.2021 17:25:59,913 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry2-2d8e79dc-f788-4286-9c64-ec6210a1ff21-StreamThread-3-consumer, groupId=staging-message-processor-retry2] Polling for fetches with timeout 96 02.12.2021 17:25:59,961 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-3-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 396 02.12.2021 17:25:59,962 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-3-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 86 02.12.2021 17:25:59,972 DEBUG [com.sigfox.commons.kafka.DelayedProcessor:39] Current message is at 239908ms for 240000ms of delay, sleeping for 102ms 02.12.2021 17:25:59,988 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry3-912326ee-6960-44d6-b637-f6bf4b4fb4d9-StreamThread-2-consumer, groupId=staging-message-processor-retry3] Polling for fetches with timeout 294 02.12.2021 17:25:59,994 TRACE [org.apache.kafka.clients.consumer.KafkaConsumer:1289] [Consumer clientId=staging-message-processor-retry1-36a5b1e4-112e-4cf6-8392-e2f5319e8176-StreamThread-2-consumer, groupId=staging-message-processor-retry1] Polling for fetches with timeout 400