I0115 03:39:03.175988 15 log.go:34] Block cache metrics: hit: 53581494 miss: 38098398 keys-added: 8431972 keys-updated: 94096 keys-evicted: 7165935 cost-added: 36865088106 cost-evicted: 31347062552 sets-dropped: 0 sets-rejected: 12743505 gets-dropped: 69888 gets-kept: 63744704 gets-total: 91679892 hit-ratio: 0.58 I0115 03:44:03.175965 15 log.go:34] Block cache metrics: hit: 53589105 miss: 38100469 keys-added: 8432213 keys-updated: 94128 keys-evicted: 7166174 cost-added: 36866127658 cost-evicted: 31348102086 sets-dropped: 0 sets-rejected: 12745303 gets-dropped: 69888 gets-kept: 63753792 gets-total: 91689574 hit-ratio: 0.58 I0115 03:49:03.176006 15 log.go:34] Block cache metrics: hit: 53589130 miss: 38100521 keys-added: 8432217 keys-updated: 94128 keys-evicted: 7166178 cost-added: 36866145514 cost-evicted: 31348119258 sets-dropped: 0 sets-rejected: 12745351 gets-dropped: 69888 gets-kept: 63753792 gets-total: 91689651 hit-ratio: 0.58 I0115 03:54:03.176086 15 log.go:34] Block cache metrics: hit: 53589392 miss: 38100639 keys-added: 8432229 keys-updated: 94128 keys-evicted: 7166190 cost-added: 36866197351 cost-evicted: 31348171919 sets-dropped: 0 sets-rejected: 12745457 gets-dropped: 69888 gets-kept: 63753984 gets-total: 91690031 hit-ratio: 0.58 I0115 03:55:57.833551 15 log.go:34] [2] [E] LOG Compact 0->2 (5, 5 -> 38 tables with 2 splits). [08198 08199 08200 08201 08202 . 08001 08002 08004 08006 08012 .] -> [08203 08205 08206 08208 08210 08212 08213 08214 08216 08218 08220 08221 08223 08225 08226 08227 08228 08229 08230 08231 08232 08233 08234 08235 08236 08237 08238 08239 08240 08204 08207 08209 08211 08215 08217 08219 08222 08224 .], took 9.692s I0115 03:56:03.178327 15 log.go:34] [0] [E] LOG Compact 3->4 (1, 3 -> 3 tables with 1 splits). [08117 . 08292 08293 08294 .] -> [08303 08313 08314 .], took 2.895s I0115 03:56:03.184258 15 log.go:34] Found value log max discard fid: 1964 discard: 1054437230 I0115 03:56:03.184294 15 log.go:34] Rewriting fid: 1964 I0115 03:56:03.762406 15 draft.go:606] Creating snapshot at Index: 92141414, ReadTs: 101505499 I0115 03:56:04.541573 15 log.go:34] [3] [E] LOG Compact 3->4 (1, 1 -> 2 tables with 1 splits). [08245 . 08305 .] -> [08317 08320 .], took 2.284s I0115 03:56:05.579332 15 log.go:34] [2] [E] LOG Compact 3->4 (1, 1 -> 2 tables with 1 splits). [08252 . 08296 .] -> [08321 08323 .], took 2.073s I0115 03:56:06.480534 15 log.go:34] [0] [E] LOG Compact 3->4 (1, 1 -> 1 tables with 1 splits). [08248 . 08183 .] -> [08324 .], took 2.256s I0115 03:56:06.593197 15 log.go:34] [3] [E] LOG Compact 4->5 (1, 0 -> 1 tables with 1 splits). [08165 . .] -> [08325 .], took 2.051s I0115 03:56:07.534582 15 log.go:34] [1] [E] LOG Compact 4->5 (1, 4 -> 3 tables with 2 splits). [07922 . 07864 07869 07801 07901 .] -> [08322 08328 08319 .], took 5.461s I0115 03:56:08.450967 15 log.go:34] [2] [E] LOG Compact 3->4 (1, 4 -> 5 tables with 2 splits). [08255 . 08118 08122 08125 08321 .] -> [08327 08331 08333 08329 08330 .], took 2.871s I0115 03:56:10.225505 15 log.go:34] [0] [E] LOG Compact 3->4 (1, 3 -> 3 tables with 1 splits). [08241 . 08128 08130 08184 .] -> [08332 08340 08343 .], took 3.588s I0115 03:56:11.598282 15 log.go:34] [1] [E] LOG Compact 3->4 (1, 3 -> 2 tables with 1 splits). [08254 . 08317 08320 08315 .] -> [08337 08344 .], took 4.064s I0115 03:56:14.656658 15 log.go:34] [2] [E] LOG Compact 4->5 (1, 4 -> 3 tables with 2 splits). [07896 . 08300 08182 07447 06673 .] -> [08339 08350 08357 .], took 6.205s I0115 03:56:15.180370 15 log.go:34] [1] [E] LOG Compact 3->4 (1, 5 -> 4 tables with 2 splits). [08263 . 08340 08343 08187 08144 08147 .] -> [08352 08361 08351 08362 .], took 3.571s I0115 03:56:15.222082 15 log.go:34] [0] [E] LOG Compact 3->4 (1, 3 -> 3 tables with 1 splits). [08257 . 08154 08155 08180 .] -> [08347 08359 08363 .], took 4.996s 115 03:56:15.751648 15 log.go:34] [3] [E] LOG Compact 3->4 (1, 12 -> 13 tables with 4 splits). [08250 . 08107 08120 08126 08135 08108 08119 08134 08106 08116 08124 08109 08327 .] -> [08348 08356 08365 08345 08353 08358 08349 08355 08360 08346 08354 08367 08368 .], took 5.55s I0115 03:56:18.878054 15 log.go:34] Processed 9 entries in 1 loops I0115 03:56:18.878084 15 log.go:34] Total entries: 3364. Moved: 78 I0115 03:56:18.878090 15 log.go:34] Removing fid: 1964 I0115 03:56:19.076786 15 log.go:34] Found value log max discard fid: 1944 discard: 1064957157 I0115 03:56:19.076833 15 log.go:34] Rewriting fid: 1944 I0115 03:56:35.461583 15 log.go:34] Processed 71 entries in 1 loops I0115 03:56:35.461605 15 log.go:34] Total entries: 3467. Moved: 71 I0115 03:56:35.461610 15 log.go:34] Removing fid: 1944 I0115 03:56:35.662339 15 log.go:34] Found value log max discard fid: 1962 discard: 1053192640 I0115 03:56:35.662374 15 log.go:34] Rewriting fid: 1962 I0115 03:56:46.877265 15 log.go:34] Processed 4 entries in 1 loops I0115 03:56:46.885746 15 log.go:34] Total entries: 3346. Moved: 73 I0115 03:56:46.886279 15 log.go:34] Removing fid: 1962 I0115 03:56:46.886723 15 log.go:34] Found value log max discard fid: 1970 discard: 1047164358 I0115 03:56:46.887010 15 log.go:34] Rewriting fid: 1970 I0115 03:57:03.137364 15 log.go:34] Processed 25 entries in 1 loops I0115 03:57:03.137583 15 log.go:34] Total entries: 3360. Moved: 93 I0115 03:57:03.137902 15 log.go:34] Removing fid: 1970 I0115 03:57:03.878092 15 log.go:34] Found value log max discard fid: 1945 discard: 1043329902 I0115 03:57:03.878582 15 log.go:34] Rewriting fid: 1945 I0115 03:57:19.474240 15 log.go:34] Processed 2 entries in 1 loops I0115 03:57:19.474806 15 log.go:34] Total entries: 3454. Moved: 127 I0115 03:57:19.475153 15 log.go:34] Removing fid: 1945 I0115 03:57:19.475488 15 log.go:34] Found value log max discard fid: 1963 discard: 1033338366 I0115 03:57:19.475718 15 log.go:34] Rewriting fid: 1963 I0115 03:57:38.040923 15 log.go:34] Processed 3 entries in 1 loops I0115 03:57:38.041139 15 log.go:34] Total entries: 3335. Moved: 131 I0115 03:57:38.041455 15 log.go:34] Removing fid: 1963 I0115 03:57:38.041726 15 log.go:34] Found value log max discard fid: 1953 discard: 1027022099 I0115 03:57:38.041945 15 log.go:34] Rewriting fid: 1953 I0115 03:57:55.758266 15 log.go:34] Processed 17 entries in 1 loops I0115 03:57:55.758296 15 log.go:34] Total entries: 3391. Moved: 159 I0115 03:57:55.758308 15 log.go:34] Removing fid: 1953 I0115 03:57:56.064365 15 log.go:34] Found value log max discard fid: 1966 discard: 1023498618 I0115 03:57:56.064409 15 log.go:34] Rewriting fid: 1966 I0115 03:58:03.708601 15 draft.go:606] Creating snapshot at Index: 92154869, ReadTs: 101532104 I0115 03:58:12.823081 15 log.go:34] Processed 7 entries in 1 loops I0115 03:58:12.823107 15 log.go:34] Total entries: 3337. Moved: 169 I0115 03:58:12.823115 15 log.go:34] Removing fid: 1966 I0115 03:58:13.320573 15 log.go:34] Found value log max discard fid: 1971 discard: 1022765085 I0115 03:58:13.320646 15 log.go:34] Rewriting fid: 1971 I0115 03:58:27.253433 15 log.go:34] Processed 4 entries in 1 loops I0115 03:58:27.253454 15 log.go:34] Total entries: 3362. Moved: 184 I0115 03:58:27.253459 15 log.go:34] Removing fid: 1971 I0115 03:58:27.432230 15 log.go:34] Found value log max discard fid: 1968 discard: 1022111099 I0115 03:58:27.432286 15 log.go:34] Rewriting fid: 1968 I0115 03:58:38.903288 15 log.go:34] Processed 7 entries in 1 loops I0115 03:58:38.911718 15 log.go:34] Total entries: 3457. Moved: 171 I0115 03:58:38.912081 15 log.go:34] Removing fid: 1968 I0115 03:58:38.912387 15 log.go:34] Found value log max discard fid: 1977 discard: 1018858017 I0115 03:58:38.912498 15 log.go:34] Rewriting fid: 1977 I0115 03:58:43.743953 15 log.go:34] Processed 9 entries in 1 loops I0115 03:58:43.744445 15 log.go:34] Total entries: 3358. Moved: 176 I0115 03:58:43.744714 15 log.go:34] Removing fid: 1977 I0115 03:58:44.512861 15 log.go:34] Found value log max discard fid: 1972 discard: 1016083438 I0115 03:58:44.513396 15 log.go:34] Rewriting fid: 1972 I0115 03:58:56.061153 15 log.go:34] Processed 3 entries in 1 loops I0115 03:58:56.061177 15 log.go:34] Total entries: 3352. Moved: 175 I0115 03:58:56.061185 15 log.go:34] Removing fid: 1972 I0115 03:58:56.402892 15 log.go:34] Found value log max discard fid: 1973 discard: 1013400969 I0115 03:58:56.402917 15 log.go:34] Rewriting fid: 1973 I0115 03:59:03.177640 15 log.go:34] Block cache metrics: hit: 54728346 miss: 38736049 keys-added: 8479247 keys-updated: 132213 keys-evicted: 7212516 cost-added: 37069720872 cost-evicted: 31551697561 sets-dropped: 0 sets-rejected: 12821856 gets-dropped: 146432 gets-kept: 65376256 gets-total: 93464395 hit-ratio: 0.59 I0115 03:59:13.086427 15 log.go:34] Processed 21 entries in 1 loops I0115 03:59:13.086680 15 log.go:34] Total entries: 3364. Moved: 192 I0115 03:59:13.086789 15 log.go:34] Removing fid: 1973 I0115 03:59:13.086996 15 log.go:34] Found value log max discard fid: 1954 discard: 1010970627 I0115 03:59:13.088891 15 log.go:34] Rewriting fid: 1954 I0115 03:59:26.878259 15 log.go:34] Processed 12 entries in 1 loops I0115 03:59:26.878610 15 log.go:34] Total entries: 5639. Moved: 2072 I0115 03:59:26.878628 15 log.go:34] Removing fid: 1954 I0115 03:59:27.137462 15 log.go:34] Found value log max discard fid: 1943 discard: 1006747493 I0115 03:59:27.137507 15 log.go:34] Rewriting fid: 1943 I0115 03:59:39.377618 15 log.go:34] Processed 74 entries in 1 loops I0115 03:59:39.377645 15 log.go:34] Total entries: 3546. Moved: 304 I0115 03:59:39.377653 15 log.go:34] Removing fid: 1943 I0115 03:59:39.559287 15 log.go:34] Found value log max discard fid: 1978 discard: 998862651 I0115 03:59:39.559324 15 log.go:34] Rewriting fid: 1978 I0115 03:59:43.613050 15 log.go:34] Processed 21 entries in 1 loops I0115 03:59:43.615856 15 log.go:34] Total entries: 3343. Moved: 243 I0115 03:59:43.617008 15 log.go:34] Removing fid: 1978 I0115 03:59:43.618022 15 log.go:34] Found value log max discard fid: 1937 discard: 996820265 I0115 03:59:43.618286 15 log.go:34] Rewriting fid: 1937 I0115 03:59:58.727917 15 log.go:34] Processed 77 entries in 1 loops I0115 03:59:58.742150 15 log.go:34] Total entries: 4163. Moved: 741 I0115 03:59:58.742181 15 log.go:34] Removing fid: 1937 I0115 03:59:58.742238 15 log.go:34] Found value log max discard fid: 1969 discard: 996589222 I0115 03:59:58.742275 15 log.go:34] Rewriting fid: 1969 I0115 04:00:03.786255 15 draft.go:606] Creating snapshot at Index: 92174749, ReadTs: 101572782 I0115 04:00:12.862393 15 log.go:34] Processed 22 entries in 1 loops I0115 04:00:12.862607 15 log.go:34] Total entries: 3372. Moved: 215 I0115 04:00:12.864744 15 log.go:34] Removing fid: 1969 I0115 04:00:12.864968 15 log.go:34] Found value log max discard fid: 1967 discard: 994430344 I0115 04:00:12.865078 15 log.go:34] Rewriting fid: 1967 I0115 04:00:29.199466 15 log.go:34] Processed 29 entries in 1 loops I0115 04:00:29.199635 15 log.go:34] Total entries: 3344. Moved: 253 I0115 04:00:29.202705 15 log.go:34] Removing fid: 1967 I0115 04:00:29.713126 15 log.go:34] Found value log max discard fid: 1936 discard: 988946907 I0115 04:00:29.713161 15 log.go:34] Rewriting fid: 1936 I0115 04:00:44.813521 15 log.go:34] Processed 49 entries in 1 loops I0115 04:00:44.813549 15 log.go:34] Total entries: 3917. Moved: 590 I0115 04:00:44.813557 15 log.go:34] Removing fid: 1936 I0115 04:00:45.034154 15 log.go:34] Found value log max discard fid: 1938 discard: 987987628 I0115 04:00:45.034753 15 log.go:34] Rewriting fid: 1938 I0115 04:00:58.859929 15 log.go:34] Processed 50 entries in 1 loops I0115 04:00:58.859956 15 log.go:34] Total entries: 4960. Moved: 1345 I0115 04:00:58.859964 15 log.go:34] Removing fid: 1938 I0115 04:00:58.860014 15 log.go:34] Found value log max discard fid: 1975 discard: 982252856 I0115 04:00:58.860030 15 log.go:34] Rewriting fid: 1975 I0115 04:01:05.170625 15 log.go:34] Processed 24 entries in 1 loops I0115 04:01:05.171159 15 log.go:34] Total entries: 3358. Moved: 290 I0115 04:01:05.171255 15 log.go:34] Removing fid: 1975 I0115 04:01:05.962827 15 log.go:34] Found value log max discard fid: 1941 discard: 979128659 I0115 04:01:05.963184 15 log.go:34] Rewriting fid: 1941 I0115 04:01:21.018776 15 log.go:34] Processed 19 entries in 1 loops I0115 04:01:21.018799 15 log.go:34] Total entries: 3851. Moved: 602 I0115 04:01:21.018804 15 log.go:34] Removing fid: 1941 I0115 04:02:04.232471 15 draft.go:606] Creating snapshot at Index: 92191872, ReadTs: 101605983 I0115 04:04:03.175975 15 log.go:34] Block cache metrics: hit: 55842718 miss: 38822781 keys-added: 8532895 keys-updated: 159766 keys-evicted: 7266001 cost-added: 37302153478 cost-evicted: 31784129563 sets-dropped: 0 sets-rejected: 12827387 gets-dropped: 171456 gets-kept: 66465088 gets-total: 94665499 hit-ratio: 0.59 I0115 04:05:03.350495 15 draft.go:606] Creating snapshot at Index: 92204415, ReadTs: 101624976 I0115 04:09:03.175971 15 log.go:34] Block cache metrics: hit: 56299603 miss: 38848652 keys-added: 8552546 keys-updated: 163207 keys-evicted: 7285572 cost-added: 37387209887 cost-evicted: 31869184630 sets-dropped: 0 sets-rejected: 12830166 gets-dropped: 171584 gets-kept: 66895744 gets-total: 95148255 hit-ratio: 0.59 I0115 04:09:03.344109 15 draft.go:606] Creating snapshot at Index: 92216759, ReadTs: 101640447 I0115 04:13:03.420371 15 draft.go:606] Creating snapshot at Index: 92229358, ReadTs: 101656295 I0115 04:14:03.176020 15 log.go:34] Block cache metrics: hit: 56733911 miss: 38873280 keys-added: 8570750 keys-updated: 166617 keys-evicted: 7303721 cost-added: 37466071459 cost-evicted: 31948049366 sets-dropped: 0 sets-rejected: 12833180 gets-dropped: 171712 gets-kept: 67293376 gets-total: 95607191 hit-ratio: 0.59 I0115 04:18:03.333981 15 draft.go:606] Creating snapshot at Index: 92241756, ReadTs: 101670596 I0115 04:19:03.175948 15 log.go:34] Block cache metrics: hit: 57163498 miss: 38897436 keys-added: 8588383 keys-updated: 169651 keys-evicted: 7321301 cost-added: 37542432023 cost-evicted: 32024409893 sets-dropped: 0 sets-rejected: 12836668 gets-dropped: 171840 gets-kept: 67702080 gets-total: 96060934 hit-ratio: 0.60 I0115 04:24:03.176418 15 log.go:34] Block cache metrics: hit: 57296948 miss: 38927071 keys-added: 8613026 keys-updated: 170536 keys-evicted: 7344934 cost-added: 37645142157 cost-evicted: 32127119167 sets-dropped: 0 sets-rejected: 12840775 gets-dropped: 179264 gets-kept: 67847680 gets-total: 96224019 hit-ratio: 0.60 I0115 04:29:03.176225 15 log.go:34] Block cache metrics: hit: 57298192 miss: 38928055 keys-added: 8613834 keys-updated: 170538 keys-evicted: 7345706 cost-added: 37648490822 cost-evicted: 32130466729 sets-dropped: 0 sets-rejected: 12840949 gets-dropped: 179264 gets-kept: 67849472 gets-total: 96226247 hit-ratio: 0.60 I0115 04:31:04.034919 15 draft.go:606] Creating snapshot at Index: 92257502, ReadTs: 101697349 I0115 04:33:03.524299 15 draft.go:606] Creating snapshot at Index: 92270705, ReadTs: 101726467 W0115 04:33:39.191426 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 522ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 3. MustSync: true I0115 04:34:03.175965 15 log.go:34] Block cache metrics: hit: 58140268 miss: 39065066 keys-added: 8695032 keys-updated: 201528 keys-evicted: 7426865 cost-added: 38001035982 cost-evicted: 32483013255 sets-dropped: 0 sets-rejected: 12865771 gets-dropped: 225024 gets-kept: 68777152 gets-total: 97205334 hit-ratio: 0.60 I0115 04:35:03.583939 15 draft.go:606] Creating snapshot at Index: 92283507, ReadTs: 101752766 I0115 04:37:07.201850 15 draft.go:606] Creating snapshot at Index: 92298098, ReadTs: 101784039 W0115 04:37:18.267911 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 602ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 04:39:03.179326 15 log.go:34] Block cache metrics: hit: 58832988 miss: 39309961 keys-added: 8842041 keys-updated: 214832 keys-evicted: 7574083 cost-added: 38640772664 cost-evicted: 33122747476 sets-dropped: 0 sets-rejected: 12950352 gets-dropped: 261824 gets-kept: 69675264 gets-total: 98142949 hit-ratio: 0.60 I0115 04:39:03.453378 15 draft.go:606] Creating snapshot at Index: 92310998, ReadTs: 101809234 W0115 04:40:32.542455 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 540ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 04:44:03.179324 15 log.go:34] Block cache metrics: hit: 59016241 miss: 39372803 keys-added: 8875767 keys-updated: 216881 keys-evicted: 7607865 cost-added: 38787596768 cost-evicted: 33269572418 sets-dropped: 0 sets-rejected: 12977421 gets-dropped: 264448 gets-kept: 69917952 gets-total: 98389044 hit-ratio: 0.60 I0115 04:46:06.021274 15 draft.go:606] Creating snapshot at Index: 92322854, ReadTs: 101829645 I0115 04:49:03.176806 15 log.go:34] Block cache metrics: hit: 59184834 miss: 39442669 keys-added: 8910873 keys-updated: 220778 keys-evicted: 7642977 cost-added: 38940249541 cost-evicted: 33422223954 sets-dropped: 0 sets-rejected: 13008284 gets-dropped: 286592 gets-kept: 70129536 gets-total: 98627503 hit-ratio: 0.60 I0115 04:51:04.007417 15 draft.go:606] Creating snapshot at Index: 92336893, ReadTs: 101862615 W0115 04:52:10.144027 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 611ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true W0115 04:53:33.706765 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 524ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true I0115 04:54:03.188452 15 log.go:34] Block cache metrics: hit: 59503339 miss: 39484805 keys-added: 8922619 keys-updated: 235070 keys-evicted: 7654713 cost-added: 38991259665 cost-evicted: 33473233373 sets-dropped: 0 sets-rejected: 13024381 gets-dropped: 301504 gets-kept: 70469376 gets-total: 98988144 hit-ratio: 0.60 W0115 04:54:48.050741 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 515ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 04:54:48.615563 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 554ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true W0115 04:55:39.125671 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 570ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 2. MustSync: true I0115 04:59:03.189848 15 log.go:34] Block cache metrics: hit: 59876619 miss: 39526562 keys-added: 8934957 keys-updated: 245881 keys-evicted: 7667021 cost-added: 39044757700 cost-evicted: 33526733377 sets-dropped: 0 sets-rejected: 13042989 gets-dropped: 314560 gets-kept: 70867392 gets-total: 99403181 hit-ratio: 0.60 W0115 05:00:24.300129 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 671ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:00:25.251221 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 568ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:00:25.900867 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 568ms. Breakdown: [{proposals 11ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:00:27.079471 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 871ms. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false I0115 05:01:08.553396 15 log.go:34] f [logterm: 9, index: 92346984, vote: f] cast MsgPreVote for 5 [logterm: 9, index: 92346984] at term 9 I0115 05:01:08.556483 15 log.go:34] f [logterm: 9, index: 92346984, vote: f] cast MsgPreVote for 4 [logterm: 9, index: 92346984] at term 9 I0115 05:01:08.556996 15 log.go:34] f [term: 9] received a MsgVote message with higher term from 5 [term: 10] I0115 05:01:09.049085 15 log.go:34] f became follower at term 10 I0115 05:01:09.132867 15 log.go:34] f [logterm: 9, index: 92346984, vote: 0] cast MsgVote for 5 [logterm: 9, index: 92346984] at term 10 I0115 05:01:09.142868 15 log.go:34] raft.node: f lost leader f at term 10 I0115 05:01:09.142923 15 log.go:34] raft.node: f elected leader 5 at term 10 W0115 05:01:09.210085 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 650ms. Breakdown: [{advance 647ms} {disk 3ms} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:01:10.644864 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 732ms. Breakdown: [{disk 715ms} {proposals 1ms} {advance 0s}] Num entries: 1. MustSync: true W0115 05:01:12.848216 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.15s. Breakdown: [{proposals 876ms} {disk 59ms} {advance 0s}] Num entries: 1. MustSync: true I0115 05:01:16.958171 15 draft.go:606] Creating snapshot at Index: 92346984, ReadTs: 101884062 E0115 05:01:18.097686 15 groups.go:1000] No longer the leader of group 2. Exiting E0115 05:01:18.123299 15 groups.go:937] Error in oracle delta stream. Error: rpc error: code = Canceled desc = context canceled W0115 05:01:48.647978 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 721ms. Breakdown: [{proposals 71ms} {disk 11ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:50.837706 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 789ms. Breakdown: [{proposals 784ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:53.022359 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.083s. Breakdown: [{proposals 600ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:01:53.607074 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 531ms. Breakdown: [{disk 526ms} {proposals 3ms} {advance 2ms}] Num entries: 1. MustSync: true W0115 05:01:56.310018 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 638ms. Breakdown: [{proposals 25ms} {disk 10ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:02:03.451523 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 512ms. Breakdown: [{proposals 506ms} {disk 1ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:02:50.583697 15 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:02:50.770946 15 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:02:51.042990 15 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:02:51.386811 15 groups.go:838] Received first state update from Zero: counter:36743258 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737090000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:04:03.196190 15 log.go:34] Block cache metrics: hit: 60006691 miss: 39537320 keys-added: 8938163 keys-updated: 248056 keys-evicted: 7670219 cost-added: 39058661365 cost-evicted: 33540635096 sets-dropped: 0 sets-rejected: 13048366 gets-dropped: 318016 gets-kept: 71001920 gets-total: 99544011 hit-ratio: 0.60 W0115 05:05:45.339240 15 groups.go:869] No membership update for 10s. Closing connection to Zero. I0115 05:05:48.987092 15 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 W0115 05:05:49.624087 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 11.568s. Breakdown: [{disk 1.902s} {advance 1.855s} {proposals 29ms}] Num entries: 0. MustSync: false E0115 05:06:13.525853 15 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:06:33.753325 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 25.897s. Breakdown: [{advance 19.76s} {proposals 5.905s} {disk 7ms}] Num entries: 0. MustSync: false I0115 05:07:06.268457 15 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:07:06.937859 15 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:08:33.212194 15 raft_server.go:239] Error while raft.Step from 0x5: context deadline exceeded. Closing RaftMessage stream. I0115 05:09:20.514202 15 log.go:34] Block cache metrics: hit: 60059808 miss: 39541110 keys-added: 8939241 keys-updated: 248811 keys-evicted: 7671294 cost-added: 39063335829 cost-evicted: 33545311013 sets-dropped: 0 sets-rejected: 13050323 gets-dropped: 321280 gets-kept: 71055552 gets-total: 99600918 hit-ratio: 0.60 W0115 05:15:00.291892 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 5m4.783s. Breakdown: [{proposals 3m29.854s} {disk 1m34.928s} {advance 0s}] Num entries: 1. MustSync: true I0115 05:16:15.094452 15 log.go:34] Block cache metrics: hit: 60059808 miss: 39541110 keys-added: 8939241 keys-updated: 248811 keys-evicted: 7671294 cost-added: 39063335829 cost-evicted: 33545311013 sets-dropped: 0 sets-rejected: 13050323 gets-dropped: 321280 gets-kept: 71055552 gets-total: 99600918 hit-ratio: 0.60 I0115 05:09:39.672717 15 groups.go:838] Received first state update from Zero: counter:36743384 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: I0115 05:21:25.550784 15 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:21:26.334435 15 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. W0115 05:21:27.804086 15 server_state.go:205] Error while retrieving timestamps: rpc error: code = DeadlineExceeded desc = context deadline exceeded with delay: 10ms. Will retry... I0115 05:21:27.947917 15 groups.go:838] Received first state update from Zero: counter:36743385 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:21:31.033644 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 932ms. Breakdown: [{proposals 501ms} {disk 417ms} {advance 14ms}] Num entries: 0. MustSync: false W0115 05:21:45.873730 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 14.107s. Breakdown: [{proposals 1ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:22:17.170313 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 806ms. Breakdown: [{proposals 802ms} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:22:30.681115 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.109s. Breakdown: [{disk 2.109s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:22:36.171126 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.42s. Breakdown: [{proposals 306ms} {disk 109ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:22:45.447796 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 566ms. Breakdown: [{disk 464ms} {proposals 101ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:22:48.396447 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.654s. Breakdown: [{proposals 1.435s} {disk 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:23:57.005949 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 6.701s. Breakdown: [{disk 972ms} {advance 748ms} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:24:02.739622 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.778s. Breakdown: [{disk 1.778s} {proposals 0s} {advance 0s}] Num entries: 1. MustSync: true W0115 05:24:05.192764 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.596s. Breakdown: [{proposals 1.378s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:09.640980 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 502ms. Breakdown: [{disk 354ms} {proposals 1ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:24:14.412796 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.558s. Breakdown: [{proposals 1.089s} {disk 24ms} {advance 0s}] Num entries: 0. MustSync: false I0115 05:24:33.269794 15 draft.go:927] Blocked pushing to raft node applyCh for 1.401s W0115 05:24:39.594136 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 7.883s. Breakdown: [{proposals 3.53s} {advance 1.084s} {disk 3ms}] Num entries: 1. MustSync: true W0115 05:24:51.297324 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.104s. Breakdown: [{advance 1.764s} {disk 1.339s} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:24:58.647848 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 4.029s. Breakdown: [{disk 4.021s} {proposals 8ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:25:02.707139 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.144s. Breakdown: [{disk 2.259s} {advance 7ms} {proposals 1ms}] Num entries: 1. MustSync: true I0115 05:25:06.339746 15 log.go:34] Block cache metrics: hit: 60062264 miss: 39542425 keys-added: 8939450 keys-updated: 249516 keys-evicted: 7671502 cost-added: 39064240979 cost-evicted: 33546216650 sets-dropped: 0 sets-rejected: 13050724 gets-dropped: 321280 gets-kept: 71055936 gets-total: 99604689 hit-ratio: 0.60 W0115 05:25:10.207656 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1s. Breakdown: [{disk 999ms} {proposals 1ms} {advance 0s}] Num entries: 1. MustSync: true W0115 05:25:27.400827 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.017s. Breakdown: [{proposals 1.015s} {disk 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:26:43.249123 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 39.89s. Breakdown: [{advance 8.447s} {proposals 10ms} {disk 0s}] Num entries: 0. MustSync: false E0115 05:27:41.008285 15 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:28:26.533435 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 878ms. Breakdown: [{proposals 498ms} {disk 374ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:29:17.351314 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 637ms. Breakdown: [{disk 594ms} {proposals 17ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:29:49.328028 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 8.742s. Breakdown: [{disk 2ms} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:29:54.231189 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 3.657s. Breakdown: [{disk 2.566s} {advance 1.09s} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:29:58.708218 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 900ms. Breakdown: [{disk 883ms} {proposals 11ms} {advance 0s}] Num entries: 1. MustSync: true I0115 05:30:04.224119 15 log.go:34] Block cache metrics: hit: 60062267 miss: 39542439 keys-added: 8939455 keys-updated: 249516 keys-evicted: 7671507 cost-added: 39064262684 cost-evicted: 33546238227 sets-dropped: 0 sets-rejected: 13050733 gets-dropped: 321280 gets-kept: 71055936 gets-total: 99604706 hit-ratio: 0.60 W0115 05:30:04.240616 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.833s. Breakdown: [{disk 0s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:30:06.965742 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.787s. Breakdown: [{disk 1.774s} {proposals 5ms} {advance 4ms}] Num entries: 1. MustSync: true W0115 05:30:49.287894 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 606ms. Breakdown: [{disk 605ms} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:30:55.287045 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 507ms. Breakdown: [{proposals 473ms} {disk 16ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:30:56.477297 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.183s. Breakdown: [{proposals 1.074s} {disk 109ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:31:21.760770 15 groups.go:869] No membership update for 10s. Closing connection to Zero. W0115 05:32:16.390872 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 57.38s. Breakdown: [{proposals 10.482s} {advance 8.563s} {disk 3.795s}] Num entries: 0. MustSync: false E0115 05:32:16.574420 15 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded I0115 05:32:16.978276 15 groups.go:807] Got address of a Zero leader: 192.168.3.11:5080 I0115 05:32:16.978491 15 groups.go:821] Starting a new membership stream receive from 192.168.3.11:5080. I0115 05:32:17.048976 15 groups.go:838] Received first state update from Zero: counter:36743390 groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101868742 checksum:18290414373100373824 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101884062 checksum:2757678515920796499 > > groups: > members: > members: > tablets: > tablets: > tablets: > tablets: > tablets: > tablets: > snapshot_ts:101872755 checksum:287855597640554238 > > zeros: > zeros: > zeros: > maxLeaseId:737100000 maxTxnTs:101900000 maxRaftId:17 removed: removed: removed: removed: removed: removed: removed: removed: removed: cid:"1582cefb-03ec-4014-b336-a4857ccc1344" license: W0115 05:32:25.221712 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 1.201s. Breakdown: [{disk 1.2s} {proposals 0s} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:27.079850 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 765ms. Breakdown: [{disk 729ms} {proposals 16ms} {advance 0s}] Num entries: 0. MustSync: false W0115 05:32:38.014691 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 2.79s. Breakdown: [{disk 2.52s} {proposals 168ms} {advance 1ms}] Num entries: 0. MustSync: false E0115 05:32:46.999685 15 groups.go:773] While sending membership update: rpc error: code = DeadlineExceeded desc = context deadline exceeded W0115 05:32:54.060787 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 695ms. Breakdown: [{advance 686ms} {disk 0s} {proposals 0s}] Num entries: 0. MustSync: false W0115 05:33:02.642022 15 draft.go:1313] Raft.Ready took too long to process: Timer Total: 717ms. Breakdown: [{disk 714ms} {proposals 2ms} {advance 0s}] Num entries: 0. MustSync: false [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:33:57 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:33:57 Integration installed: Environment [Sentry] 2021/01/15 05:33:57 Integration installed: Modules [Sentry] 2021/01/15 05:33:57 Integration installed: IgnoreErrors [Decoder]: Using assembly version of decoder Page Size: 4096 [Sentry] 2021/01/15 05:33:57 Integration installed: ContextifyFrames [Sentry] 2021/01/15 05:33:57 Integration installed: Environment [Sentry] 2021/01/15 05:33:57 Integration installed: Modules [Sentry] 2021/01/15 05:33:57 Integration installed: IgnoreErrors I0115 05:33:57.722879 15 sentry_integration.go:48] This instance of Dgraph will send anonymous reports of panics back to Dgraph Labs via Sentry. No confidential information is sent. These reports help improve Dgraph. To opt-out, restart your instance with the --enable_sentry=false flag. For more info, see https://dgraph.io/docs/howto/#data-handling. W0115 05:33:57.725251 15 run.go:573] --lru_mb is deprecated, use --cache_mb instead I0115 05:33:57.732039 15 util_ee.go:126] KeyReader instantiated of type I0115 05:33:57.971534 15 init.go:107]