Input buildinfo: https://buildinfos.debian.net/buildinfo-pool/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1_all.buildinfo Use metasnap for getting required timestamps New buildinfo file: /tmp/golang-github-hashicorp-raft-1.1.2-17hrdv2zw/golang-github-hashicorp-raft_1.1.2-1_all.buildinfo Get source package info: golang-github-hashicorp-raft=1.1.2-1 Source URL: http://snapshot.notset.fr/mr/package/golang-github-hashicorp-raft/1.1.2-1/srcfiles?fileinfo=1 env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin TMPDIR=/tmp mmdebstrap --arch=amd64 --include=autoconf=2.69-11.1 automake=1:1.16.1-4 autopoint=0.19.8.1-10 autotools-dev=20180224.1 base-files=11 base-passwd=3.5.47 bash=5.0-5 binutils=2.33.50.20200115-2 binutils-common=2.33.50.20200115-2 binutils-x86-64-linux-gnu=2.33.50.20200115-2 bsdmainutils=11.1.2+b1 bsdutils=1:2.34-0.1 build-essential=12.8 bzip2=1.0.8-2 coreutils=8.30-3+b1 cpp=4:9.2.1-3.1 cpp-9=9.2.1-24 dash=0.5.10.2-6 debconf=1.5.73 debhelper=12.8 debianutils=4.9.1 dh-autoreconf=19 dh-golang=1.45 dh-strip-nondeterminism=1.6.3-2 diffutils=1:3.7-3 dpkg=1.19.7 dpkg-dev=1.19.7 dwz=0.13-5 fdisk=2.34-0.1 file=1:5.38-4 findutils=4.7.0-1 g++=4:9.2.1-3.1 g++-9=9.2.1-24 gcc=4:9.2.1-3.1 gcc-9=9.2.1-24 gcc-9-base=9.2.1-24 gettext=0.19.8.1-10 gettext-base=0.19.8.1-10 golang-1.13-go=1.13.6-2 golang-1.13-src=1.13.6-2 golang-any=2:1.13~1 golang-github-alecthomas-units-dev=0.0~git20151022.0.2efee85-4 golang-github-armon-go-metrics-dev=0.3.0-1 golang-github-beorn7-perks-dev=1.0.1-1 golang-github-cespare-xxhash-dev=2.1.1-1 golang-github-circonus-labs-circonus-gometrics-dev=2.3.1-2 golang-github-circonus-labs-circonusllhist-dev=0.0~git20160526.0.d724266-2 golang-github-datadog-datadog-go-dev=2.1.0-2 golang-github-davecgh-go-spew-dev=1.1.1-2 golang-github-dgrijalva-jwt-go-v3-dev=3.2.0-2 golang-github-fatih-color-dev=1.7.0-1 golang-github-go-kit-kit-dev=0.6.0-2 golang-github-go-logfmt-logfmt-dev=0.3.0-1 golang-github-go-stack-stack-dev=1.5.2-2 golang-github-golang-mock-dev=1.3.1-2 golang-github-google-go-cmp-dev=0.3.1-1 golang-github-google-gofuzz-dev=0.0~git20170612.24818f7-1 golang-github-hashicorp-go-cleanhttp-dev=0.5.1-1 golang-github-hashicorp-go-hclog-dev=0.11.0-1 golang-github-hashicorp-go-immutable-radix-dev=1.1.0-1 golang-github-hashicorp-go-msgpack-dev=0.5.5-1 golang-github-hashicorp-go-retryablehttp-dev=0.6.4-1 golang-github-hashicorp-go-uuid-dev=1.0.1-2 golang-github-hashicorp-golang-lru-dev=0.5.4-1 golang-github-jpillora-backoff-dev=1.0.0-1 golang-github-json-iterator-go-dev=1.1.4-1 golang-github-julienschmidt-httprouter-dev=1.1-5 golang-github-kr-pretty-dev=0.1.0-1 golang-github-kr-pty-dev=1.1.6-1 golang-github-kr-text-dev=0.1.0-1 golang-github-mattn-go-colorable-dev=0.0.9-3 golang-github-mattn-go-isatty-dev=0.0.8-2 golang-github-modern-go-concurrent-dev=1.0.3-1 golang-github-modern-go-reflect2-dev=1.0.0-1 golang-github-mwitkow-go-conntrack-dev=0.0~git20190716.2f06839-1 golang-github-opentracing-opentracing-go-dev=1.0.2-1 golang-github-pascaldekloe-goe-dev=0.1.0-2 golang-github-pkg-errors-dev=0.8.1-1 golang-github-pmezard-go-difflib-dev=1.0.0-2 golang-github-prometheus-client-golang-dev=1.2.1-3 golang-github-prometheus-client-model-dev=0.0.2+git20171117.99fa1f4-1 golang-github-prometheus-common-dev=0.7.0-1 golang-github-sirupsen-logrus-dev=1.4.2-1 golang-github-stretchr-objx-dev=0.1.1+git20180825.ef50b0d-1 golang-github-stretchr-testify-dev=1.4.0+ds-1 golang-github-tv42-httpunix-dev=0.0~git20150427.b75d861-2 golang-github-ugorji-go-codec-dev=1.1.7-1 golang-github-ugorji-go-msgpack-dev=0.0~git20130605.792643-5 golang-github-vmihailenco-tagparser-dev=0.1.1-2 golang-glog-dev=0.0~git20160126.23def4e-3 golang-go=2:1.13~1 golang-golang-x-net-dev=1:0.0+git20191112.2180aed+dfsg-3 golang-golang-x-oauth2-dev=0.0~git20190604.0f29369-2 golang-golang-x-oauth2-google-dev=0.0~git20190604.0f29369-2 golang-golang-x-sync-dev=0.0~git20190423.1122301-1 golang-golang-x-sys-dev=0.0~git20190726.fc99dfb-1 golang-golang-x-text-dev=0.3.2-3 golang-golang-x-time-dev=0.0~git20161028.0.f51c127-2 golang-golang-x-tools-dev=1:0.0~git20191118.07fc4c7+ds-1 golang-golang-x-xerrors-dev=0.0~git20190717.a985d34-1 golang-google-cloud-compute-metadata-dev=0.43.0-1 golang-google-genproto-dev=0.0~git20190801.fa694d8-2 golang-google-grpc-dev=1.22.1-1 golang-gopkg-alecthomas-kingpin.v2-dev=2.2.6-1 golang-gopkg-check.v1-dev=0.0+git20180628.788fd78-1 golang-gopkg-mgo.v2-dev=2016.08.01-6 golang-gopkg-tomb.v2-dev=0.0~git20161208.d5d1b58-3 golang-gopkg-vmihailenco-msgpack.v2-dev=4.3.1-1 golang-gopkg-yaml.v2-dev=2.2.2-1 golang-goprotobuf-dev=1.3.2-2 golang-procfs-dev=0.0.8-1 golang-prometheus-client-dev=0.9.0-1 golang-protobuf-extensions-dev=1.0.1-1 golang-src=2:1.13~1 grep=3.3-1 groff-base=1.22.4-4 gzip=1.9-3+b1 hostname=3.23 init-system-helpers=1.57 intltool-debian=0.35.0+20060710.5 libacl1=2.2.53-5 libarchive-zip-perl=1.67-1 libasan5=9.2.1-24 libatomic1=9.2.1-24 libattr1=1:2.4.48-5 libaudit-common=1:2.8.5-2 libaudit1=1:2.8.5-2+b1 libbinutils=2.33.50.20200115-2 libblkid1=2.34-0.1 libbsd0=0.10.0-1 libbz2-1.0=1.0.8-2 libc-bin=2.29-9 libc-dev-bin=2.29-9 libc6=2.29-9 libc6-dev=2.29-9 libcap-ng0=0.7.9-2.1+b1 libcc1-0=9.2.1-24 libcroco3=0.6.13-1 libcrypt-dev=1:4.4.10-10 libcrypt1=1:4.4.10-10 libctf-nobfd0=2.33.50.20200115-2 libctf0=2.33.50.20200115-2 libdb5.3=5.3.28+dfsg1-0.6 libdebconfclient0=0.250 libdebhelper-perl=12.8 libdpkg-perl=1.19.7 libelf1=0.176-1.1 libfdisk1=2.34-0.1 libffi7=3.3-3 libfile-stripnondeterminism-perl=1.6.3-2 libgcc-9-dev=9.2.1-24 libgcc1=1:9.2.1-24 libgcrypt20=1.8.5-3 libgdbm-compat4=1.18.1-5 libgdbm6=1.18.1-5 libglib2.0-0=2.62.4-1+b1 libgmp10=2:6.1.2+dfsg-4 libgomp1=9.2.1-24 libgpg-error0=1.36-7 libicu63=63.2-2 libisl22=0.22-2 libitm1=9.2.1-24 liblsan0=9.2.1-24 liblz4-1=1.9.2-2 liblzma5=5.2.4-1+b1 libmagic-mgc=1:5.38-4 libmagic1=1:5.38-4 libmount1=2.34-0.1 libmpc3=1.1.0-1 libmpfr6=4.0.2-1 libncursesw6=6.1+20191019-1 libpam-modules=1.3.1-5 libpam-modules-bin=1.3.1-5 libpam-runtime=1.3.1-5 libpam0g=1.3.1-5 libpcre2-8-0=10.34-7 libpcre3=2:8.39-12+b1 libperl5.30=5.30.0-9 libpipeline1=1.5.2-2 libprotobuf-dev=3.6.1.3-2+b1 libprotobuf-lite17=3.6.1.3-2+b1 libprotobuf17=3.6.1.3-2+b1 libprotoc17=3.6.1.3-2+b1 libquadmath0=9.2.1-24 libsasl2-2=2.1.27+dfsg-2 libsasl2-dev=2.1.27+dfsg-2 libsasl2-modules-db=2.1.27+dfsg-2 libseccomp2=2.4.2-2 libselinux1=3.0-1 libsigsegv2=2.12-2 libsmartcols1=2.34-0.1 libstdc++-9-dev=9.2.1-24 libstdc++6=9.2.1-24 libsub-override-perl=0.09-2 libsystemd0=244-3 libtinfo6=6.1+20191019-1 libtool=2.4.6-11 libtsan0=9.2.1-24 libubsan1=9.2.1-24 libuchardet0=0.0.6-3 libudev1=244-3 libunistring2=0.9.10-2 libuuid1=2.34-0.1 libxml2=2.9.4+dfsg1-8 linux-libc-dev=5.4.13-1 login=1:4.8-1 lsb-base=11.1.0 m4=1.4.18-4 make=4.2.1-1.2 man-db=2.9.0-2 mawk=1.3.3-20 ncurses-base=6.1+20191019-1 ncurses-bin=6.1+20191019-1 patch=2.7.6-6 perl=5.30.0-9 perl-base=5.30.0-9 perl-modules-5.30=5.30.0-9 po-debconf=1.0.21 protobuf-compiler=3.6.1.3-2+b1 sed=4.7-1 sensible-utils=0.0.12+nmu1 sysvinit-utils=2.96-2.1 tar=1.30+dfsg-6+b1 util-linux=2.34-0.1 xz-utils=5.2.4-1+b1 zlib1g=1:1.2.11.dfsg-1+b1 zlib1g-dev=1:1.2.11.dfsg-1+b1 --variant=apt --aptopt=Acquire::Check-Valid-Until "false" --aptopt=Acquire::http::Dl-Limit "1000"; --aptopt=Acquire::https::Dl-Limit "1000"; --aptopt=Acquire::Retries "5"; --aptopt=APT::Get::allow-downgrades "true"; --keyring=/usr/share/keyrings/ --essential-hook=chroot "$1" sh -c "apt-get --yes install fakeroot util-linux" --essential-hook=copy-in /usr/share/keyrings/debian-archive-bullseye-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-security-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-stable.gpg /usr/share/keyrings/debian-archive-buster-automatic.gpg /usr/share/keyrings/debian-archive-buster-security-automatic.gpg /usr/share/keyrings/debian-archive-buster-stable.gpg /usr/share/keyrings/debian-archive-keyring.gpg /usr/share/keyrings/debian-archive-removed-keys.gpg /usr/share/keyrings/debian-archive-stretch-automatic.gpg /usr/share/keyrings/debian-archive-stretch-security-automatic.gpg /usr/share/keyrings/debian-archive-stretch-stable.gpg /usr/share/keyrings/debian-ports-archive-keyring-removed.gpg /usr/share/keyrings/debian-ports-archive-keyring.gpg /usr/share/keyrings/debian-keyring.gpg /etc/apt/trusted.gpg.d/ --essential-hook=chroot "$1" sh -c "rm /etc/apt/sources.list && echo 'deb http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb-src http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb http://snapshot.notset.fr/archive/debian/20200122T091601Z/ unstable main' >> /etc/apt/sources.list && apt-get update" --customize-hook=chroot "$1" useradd --no-create-home -d /nonexistent -p "" builduser -s /bin/bash --customize-hook=chroot "$1" env sh -c "apt-get source --only-source -d golang-github-hashicorp-raft=1.1.2-1 && mkdir -p /build/golang-github-hashicorp-raft-HJm3EX && dpkg-source --no-check -x /*.dsc /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && chown -R builduser:builduser /build/golang-github-hashicorp-raft-HJm3EX" --customize-hook=chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all" --customize-hook=sync-out /build/golang-github-hashicorp-raft-HJm3EX /tmp/golang-github-hashicorp-raft-1.1.2-17hrdv2zw bullseye /dev/null deb http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable main I: automatically chosen mode: root I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen format: tar I: using /tmp/mmdebstrap.RjNL4jYtQs as tempdir I: running apt-get update... I: downloading packages with apt... I: extracting archives... I: installing essential packages... I: running --essential-hook in shell: sh -c 'chroot "$1" sh -c "apt-get --yes install fakeroot util-linux"' exec /tmp/mmdebstrap.RjNL4jYtQs Reading package lists... Building dependency tree... util-linux is already the newest version (2.34-0.1). The following NEW packages will be installed: fakeroot libfakeroot 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 132 kB of archives. After this operation, 393 kB of additional disk space will be used. Get:1 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable/main amd64 libfakeroot amd64 1.24-1 [45.7 kB] Get:2 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable/main amd64 fakeroot amd64 1.24-1 [85.9 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 132 kB in 0s (756 kB/s) Selecting previously unselected package libfakeroot:amd64. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 4508 files and directories currently installed.) Preparing to unpack .../libfakeroot_1.24-1_amd64.deb ... Unpacking libfakeroot:amd64 (1.24-1) ... Selecting previously unselected package fakeroot. Preparing to unpack .../fakeroot_1.24-1_amd64.deb ... Unpacking fakeroot (1.24-1) ... Setting up libfakeroot:amd64 (1.24-1) ... Setting up fakeroot (1.24-1) ... update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode Processing triggers for libc-bin (2.29-9) ... I: running special hook: copy-in /usr/share/keyrings/debian-archive-bullseye-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-security-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-stable.gpg /usr/share/keyrings/debian-archive-buster-automatic.gpg /usr/share/keyrings/debian-archive-buster-security-automatic.gpg /usr/share/keyrings/debian-archive-buster-stable.gpg /usr/share/keyrings/debian-archive-keyring.gpg /usr/share/keyrings/debian-archive-removed-keys.gpg /usr/share/keyrings/debian-archive-stretch-automatic.gpg /usr/share/keyrings/debian-archive-stretch-security-automatic.gpg /usr/share/keyrings/debian-archive-stretch-stable.gpg /usr/share/keyrings/debian-ports-archive-keyring-removed.gpg /usr/share/keyrings/debian-ports-archive-keyring.gpg /usr/share/keyrings/debian-keyring.gpg /etc/apt/trusted.gpg.d/ I: running --essential-hook in shell: sh -c 'chroot "$1" sh -c "rm /etc/apt/sources.list && echo 'deb http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb-src http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb http://snapshot.notset.fr/archive/debian/20200122T091601Z/ unstable main' >> /etc/apt/sources.list && apt-get update"' exec /tmp/mmdebstrap.RjNL4jYtQs Get:1 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm InRelease [81.6 kB] Hit:2 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable InRelease Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Get:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources [11.4 MB] Get:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages [11.1 MB] Fetched 22.6 MB in 20s (1151 kB/s) Reading package lists... I: installing remaining packages inside the chroot... I: running --customize-hook in shell: sh -c 'chroot "$1" useradd --no-create-home -d /nonexistent -p "" builduser -s /bin/bash' exec /tmp/mmdebstrap.RjNL4jYtQs I: running --customize-hook in shell: sh -c 'chroot "$1" env sh -c "apt-get source --only-source -d golang-github-hashicorp-raft=1.1.2-1 && mkdir -p /build/golang-github-hashicorp-raft-HJm3EX && dpkg-source --no-check -x /*.dsc /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && chown -R builduser:builduser /build/golang-github-hashicorp-raft-HJm3EX"' exec /tmp/mmdebstrap.RjNL4jYtQs Reading package lists... NOTICE: 'golang-github-hashicorp-raft' packaging is maintained in the 'Git' version control system at: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Please use: git clone https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git to retrieve the latest (possibly unreleased) updates to the package. Need to get 115 kB of source archives. Get:1 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (dsc) [2408 B] Get:2 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (tar) [103 kB] Get:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (diff) [8980 B] Fetched 115 kB in 0s (721 kB/s) Download complete and in download only mode W: Download is performed unsandboxed as root as file 'golang-github-hashicorp-raft_1.1.2-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) dpkg-source: info: extracting golang-github-hashicorp-raft in /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: running --customize-hook in shell: sh -c 'chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all"' exec /tmp/mmdebstrap.RjNL4jYtQs dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build-indep dh build-indep --buildsystem=golang --with=golang dh_update_autotools_config -i -O--buildsystem=golang dh_autoreconf -i -O--buildsystem=golang dh_auto_configure -i -O--buildsystem=golang dh_auto_build -i -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/race runtime/internal/sys internal/cpu runtime/internal/atomic sync/atomic runtime/internal/math unicode/utf8 unicode internal/bytealg container/list crypto/internal/subtle crypto/subtle math/bits internal/testlog encoding runtime math unicode/utf16 internal/nettrace runtime/cgo vendor/golang.org/x/crypto/cryptobyte/asn1 vendor/golang.org/x/crypto/internal/subtle vendor/golang.org/x/crypto/curve25519 internal/reflectlite sync internal/singleflight math/rand errors sort internal/oserror io strconv syscall github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage bytes reflect bufio strings hash time internal/syscall/unix hash/crc32 github.com/hashicorp/go-immutable-radix crypto crypto/internal/randutil crypto/hmac crypto/rc4 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform path context internal/poll regexp/syntax html encoding/binary internal/fmtsort os hash/crc64 crypto/cipher encoding/base64 crypto/sha512 crypto/aes crypto/des crypto/ed25519/internal/edwards25519 fmt crypto/md5 net crypto/sha1 crypto/sha256 encoding/pem math/big log encoding/json compress/flate compress/gzip path/filepath encoding/hex net/url io/ioutil crypto/rand crypto/elliptic encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa crypto/x509/pkix crypto/ecdsa vendor/golang.org/x/crypto/cryptobyte vendor/golang.org/x/crypto/internal/chacha20 vendor/golang.org/x/crypto/poly1305 vendor/golang.org/x/sys/cpu vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal os/signal vendor/golang.org/x/net/idna golang.org/x/sys/unix regexp encoding/gob go/token crypto/x509 net/textproto vendor/golang.org/x/net/http/httpguts vendor/golang.org/x/net/http/httpproxy mime/multipart github.com/mattn/go-isatty github.com/mattn/go-colorable github.com/fatih/color text/template/parse flag github.com/hashicorp/go-hclog crypto/tls runtime/debug runtime/trace testing text/template html/template net/http/httptrace net/http github.com/armon/go-metrics net/rpc github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -i -O--buildsystem=golang cd obj-x86_64-linux-gnu && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.00s) testing.go:153: 2021-10-08T13:00:59.988Z [INFO] creating new snapshot: path=/tmp/raft278313309/raft381087512/snapshots/3-10-1633698059988.tmp === RUN TestFileSS_CreateSnapshot --- PASS: TestFileSS_CreateSnapshot (0.00s) testing.go:153: 2021-10-08T13:00:59.990Z [INFO] creating new snapshot: path=/tmp/raft689902487/snapshots/3-10-1633698059990.tmp testing.go:153: 2021-10-08T13:00:59.991Z [WARN] found temporary snapshot: name=3-10-1633698059990.tmp === RUN TestFileSS_CancelSnapshot --- PASS: TestFileSS_CancelSnapshot (0.00s) testing.go:153: 2021-10-08T13:00:59.995Z [INFO] creating new snapshot: path=/tmp/raft034278410/snapshots/3-10-1633698059995.tmp === RUN TestFileSS_Retention --- PASS: TestFileSS_Retention (0.01s) testing.go:153: 2021-10-08T13:00:59.996Z [INFO] creating new snapshot: path=/tmp/raft312783329/snapshots/3-10-1633698059996.tmp testing.go:153: 2021-10-08T13:00:59.998Z [INFO] creating new snapshot: path=/tmp/raft312783329/snapshots/3-11-1633698059998.tmp testing.go:153: 2021-10-08T13:01:00.001Z [INFO] creating new snapshot: path=/tmp/raft312783329/snapshots/3-12-1633698060001.tmp testing.go:153: 2021-10-08T13:01:00.003Z [INFO] reaping snapshot: path=/tmp/raft312783329/snapshots/3-10-1633698059996 testing.go:153: 2021-10-08T13:01:00.003Z [INFO] creating new snapshot: path=/tmp/raft312783329/snapshots/3-13-1633698060003.tmp testing.go:153: 2021-10-08T13:01:00.006Z [INFO] reaping snapshot: path=/tmp/raft312783329/snapshots/3-11-1633698059998 testing.go:153: 2021-10-08T13:01:00.006Z [INFO] creating new snapshot: path=/tmp/raft312783329/snapshots/3-14-1633698060006.tmp testing.go:153: 2021-10-08T13:01:00.008Z [INFO] reaping snapshot: path=/tmp/raft312783329/snapshots/3-12-1633698060001 === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering --- PASS: TestFileSS_Ordering (0.00s) testing.go:153: 2021-10-08T13:01:00.011Z [INFO] creating new snapshot: path=/tmp/raft967534528/snapshots/5-130350-1633698060011.tmp testing.go:153: 2021-10-08T13:01:00.013Z [INFO] creating new snapshot: path=/tmp/raft967534528/snapshots/36-204917-1633698060013.tmp === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) net_transport_test.go:339: Disabled unreliable test === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff --- PASS: TestNetworkTransport_ListenBackoff (4.00s) net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.035Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.040Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.050Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.070Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.111Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.191Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.351Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:00.671Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:01.312Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:02.312Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-08T13:01:03.312Z [ERROR] test: failed to accept connection: error="intentional error in test" === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-ab035e19-e1b0-386d-7dcb-ccfd890de16a: 2021-10-08T13:01:04.037Z [INFO] server-ab035e19-e1b0-386d-7dcb-ccfd890de16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab035e19-e1b0-386d-7dcb-ccfd890de16a Address:ab035e19-e1b0-386d-7dcb-ccfd890de16a}]" --- PASS: TestRaft_StartStop (0.00s) testing.go:149: server-ab035e19-e1b0-386d-7dcb-ccfd890de16a: 2021-10-08T13:01:04.037Z [INFO] server-ab035e19-e1b0-386d-7dcb-ccfd890de16a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ab035e19-e1b0-386d-7dcb-ccfd890de16a Address:ab035e19-e1b0-386d-7dcb-ccfd890de16a}]" === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-81d0be7c-3c48-cfad-9100-02ffce503791: 2021-10-08T13:01:04.038Z [INFO] server-81d0be7c-3c48-cfad-9100-02ffce503791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81d0be7c-3c48-cfad-9100-02ffce503791 Address:81d0be7c-3c48-cfad-9100-02ffce503791}]" --- PASS: TestRaft_AfterShutdown (0.00s) testing.go:149: server-81d0be7c-3c48-cfad-9100-02ffce503791: 2021-10-08T13:01:04.038Z [INFO] server-81d0be7c-3c48-cfad-9100-02ffce503791: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-81d0be7c-3c48-cfad-9100-02ffce503791 Address:81d0be7c-3c48-cfad-9100-02ffce503791}]" === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.040Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.040Z [INFO] server-ce97d577-bf59-ff56-a465-9d8b867f4493: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.040Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering follower state: follower="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Follower]" leader= testLoggerAdapter verbose: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.040Z [INFO] server-87f1315d-f348-ad92-3183-2fc3882b69dc: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.040Z [INFO] server-ce97d577-bf59-ff56-a465-9d8b867f4493: entering follower state: follower="Node at ce97d577-bf59-ff56-a465-9d8b867f4493 [Follower]" leader= testLoggerAdapter verbose: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.040Z [INFO] server-87f1315d-f348-ad92-3183-2fc3882b69dc: entering follower state: follower="Node at 87f1315d-f348-ad92-3183-2fc3882b69dc [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.040Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.099Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.099Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering candidate state: node="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Candidate]" term=2 testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: election won: tally=2 testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering leader state: leader="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Leader]" testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: added peer, starting replication: peer=server-ce97d577-bf59-ff56-a465-9d8b867f4493 testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: added peer, starting replication: peer=server-87f1315d-f348-ad92-3183-2fc3882b69dc testLoggerAdapter verbose: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.100Z [WARN] server-87f1315d-f348-ad92-3183-2fc3882b69dc: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.100Z [WARN] server-ce97d577-bf59-ff56-a465-9d8b867f4493: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: appendEntries rejected, sending older logs: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" next=1 testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: pipelining replication: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: appendEntries rejected, sending older logs: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" next=1 testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: pipelining replication: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.205Z [INFO] stable state for Follower reached at 2021-10-08 13:01:04.100257026 +0000 UTC m=+4.114939799 (2 nodes), 59.326359ms from start of poll, 59.807613ms from cluster start. Timeout at 2021-10-08 13:01:04.205359433 +0000 UTC m=+4.220042156, 105.102357ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.205Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.310Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.205583347 +0000 UTC m=+4.220266083 (1 nodes), 939ns from start of poll, 165.133897ms from cluster start. Timeout at 2021-10-08 13:01:04.310647073 +0000 UTC m=+4.325329802, 105.063719ms after stability testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.321Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: aborting pipeline replication: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" testLoggerAdapter verbose: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.321Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: aborting pipeline replication: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" --- PASS: TestRaft_LiveBootstrap (0.28s) testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.040Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: initial configuration: index=0 servers=[] testing.go:149: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.040Z [INFO] server-ce97d577-bf59-ff56-a465-9d8b867f4493: initial configuration: index=0 servers=[] testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.040Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering follower state: follower="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Follower]" leader= testing.go:149: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.040Z [INFO] server-87f1315d-f348-ad92-3183-2fc3882b69dc: initial configuration: index=0 servers=[] testing.go:149: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.040Z [INFO] server-ce97d577-bf59-ff56-a465-9d8b867f4493: entering follower state: follower="Node at ce97d577-bf59-ff56-a465-9d8b867f4493 [Follower]" leader= testing.go:149: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.040Z [INFO] server-87f1315d-f348-ad92-3183-2fc3882b69dc: entering follower state: follower="Node at 87f1315d-f348-ad92-3183-2fc3882b69dc [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:04.040Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.099Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.099Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering candidate state: node="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Candidate]" term=2 testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: election won: tally=2 testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: entering leader state: leader="Node at 9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5 [Leader]" testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: added peer, starting replication: peer=server-ce97d577-bf59-ff56-a465-9d8b867f4493 testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: added peer, starting replication: peer=server-87f1315d-f348-ad92-3183-2fc3882b69dc testing.go:149: server-87f1315d-f348-ad92-3183-2fc3882b69dc: 2021-10-08T13:01:04.100Z [WARN] server-87f1315d-f348-ad92-3183-2fc3882b69dc: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-ce97d577-bf59-ff56-a465-9d8b867f4493: 2021-10-08T13:01:04.100Z [WARN] server-ce97d577-bf59-ff56-a465-9d8b867f4493: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: appendEntries rejected, sending older logs: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" next=1 testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: pipelining replication: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [WARN] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: appendEntries rejected, sending older logs: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" next=1 testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.100Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: pipelining replication: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" testing.go:149: cluster: 2021-10-08T13:01:04.205Z [INFO] stable state for Follower reached at 2021-10-08 13:01:04.100257026 +0000 UTC m=+4.114939799 (2 nodes), 59.326359ms from start of poll, 59.807613ms from cluster start. Timeout at 2021-10-08 13:01:04.205359433 +0000 UTC m=+4.220042156, 105.102357ms after stability testing.go:149: cluster: 2021-10-08T13:01:04.205Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:04.310Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.205583347 +0000 UTC m=+4.220266083 (1 nodes), 939ns from start of poll, 165.133897ms from cluster start. Timeout at 2021-10-08 13:01:04.310647073 +0000 UTC m=+4.325329802, 105.063719ms after stability testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.321Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: aborting pipeline replication: peer="{Voter server-87f1315d-f348-ad92-3183-2fc3882b69dc 87f1315d-f348-ad92-3183-2fc3882b69dc}" testing.go:149: server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: 2021-10-08T13:01:04.321Z [INFO] server-9d3444b8-cc2e-bd9a-2aa2-588ed639d2c5: aborting pipeline replication: peer="{Voter server-ce97d577-bf59-ff56-a465-9d8b867f4493 ce97d577-bf59-ff56-a465-9d8b867f4493}" === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-b524928a-ee58-cd00-09eb-0a31a43cfac4: 2021-10-08T13:01:04.322Z [INFO] server-b524928a-ee58-cd00-09eb-0a31a43cfac4: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-b524928a-ee58-cd00-09eb-0a31a43cfac4: 2021-10-08T13:01:04.322Z [INFO] server-b524928a-ee58-cd00-09eb-0a31a43cfac4: entering follower state: follower="Node at b524928a-ee58-cd00-09eb-0a31a43cfac4 [Follower]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) testing.go:149: server-b524928a-ee58-cd00-09eb-0a31a43cfac4: 2021-10-08T13:01:04.322Z [INFO] server-b524928a-ee58-cd00-09eb-0a31a43cfac4: initial configuration: index=0 servers=[] testing.go:149: server-b524928a-ee58-cd00-09eb-0a31a43cfac4: 2021-10-08T13:01:04.322Z [INFO] server-b524928a-ee58-cd00-09eb-0a31a43cfac4: entering follower state: follower="Node at b524928a-ee58-cd00-09eb-0a31a43cfac4 [Follower]" leader= === RUN TestRaft_RecoverCluster --- SKIP: TestRaft_RecoverCluster (0.00s) raft_test.go:119: Disabled unreliable test === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: 2021-10-08T13:01:04.324Z [INFO] server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b}]" testLoggerAdapter verbose: server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: 2021-10-08T13:01:04.324Z [INFO] server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: entering follower state: follower="Node at 07182d5d-d9ce-987d-e7fc-1076de26d2e7 [Follower]" leader= testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.324Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b}]" testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.325Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering follower state: follower="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Follower]" leader= testLoggerAdapter verbose: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.325Z [INFO] server-5b0f090b-d63c-f47d-2644-d377258d5290: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.325Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.325Z [INFO] server-5b0f090b-d63c-f47d-2644-d377258d5290: entering follower state: follower="Node at 5b0f090b-d63c-f47d-2644-d377258d5290 [Follower]" leader= testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [WARN] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering candidate state: node="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Candidate]" term=2 testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: election won: tally=2 testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering leader state: leader="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Leader]" testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: added peer, starting replication: peer=server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: pipelining replication: peer="{Voter server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 07182d5d-d9ce-987d-e7fc-1076de26d2e7}" testLoggerAdapter verbose: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.389Z [WARN] server-5b0f090b-d63c-f47d-2644-d377258d5290: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.483Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.378607107 +0000 UTC m=+4.393289841 (1 nodes), 52.98797ms from start of poll, 54.19263ms from cluster start. Timeout at 2021-10-08 13:01:04.483707834 +0000 UTC m=+4.498390555, 105.100714ms after stability testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.483Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: updating configuration: command=AddStaging server-id=server-5b0f090b-d63c-f47d-2644-d377258d5290 server-addr=5b0f090b-d63c-f47d-2644-d377258d5290 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b} {Suffrage:Voter ID:server-5b0f090b-d63c-f47d-2644-d377258d5290 Address:5b0f090b-d63c-f47d-2644-d377258d5290}]" testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: added peer, starting replication: peer=server-5b0f090b-d63c-f47d-2644-d377258d5290 testLoggerAdapter verbose: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.484Z [WARN] server-5b0f090b-d63c-f47d-2644-d377258d5290: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [WARN] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: appendEntries rejected, sending older logs: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" next=1 testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: pipelining replication: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.489Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.594Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.489382057 +0000 UTC m=+4.504064776 (1 nodes), 720ns from start of poll, 164.967565ms from cluster start. Timeout at 2021-10-08 13:01:04.594490251 +0000 UTC m=+4.609172971, 105.108195ms after stability testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.594Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: aborting pipeline replication: peer="{Voter server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 07182d5d-d9ce-987d-e7fc-1076de26d2e7}" testLoggerAdapter verbose: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.594Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: aborting pipeline replication: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" --- PASS: TestRaft_HasExistingState (0.27s) testing.go:149: server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: 2021-10-08T13:01:04.324Z [INFO] server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b}]" testing.go:149: server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: 2021-10-08T13:01:04.324Z [INFO] server-07182d5d-d9ce-987d-e7fc-1076de26d2e7: entering follower state: follower="Node at 07182d5d-d9ce-987d-e7fc-1076de26d2e7 [Follower]" leader= testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.324Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b}]" testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.325Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering follower state: follower="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Follower]" leader= testing.go:149: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.325Z [INFO] server-5b0f090b-d63c-f47d-2644-d377258d5290: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-08T13:01:04.325Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.325Z [INFO] server-5b0f090b-d63c-f47d-2644-d377258d5290: entering follower state: follower="Node at 5b0f090b-d63c-f47d-2644-d377258d5290 [Follower]" leader= testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [WARN] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering candidate state: node="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Candidate]" term=2 testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: election won: tally=2 testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: entering leader state: leader="Node at c09f793d-0175-c5d5-9e56-821ff4cbb94b [Leader]" testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: added peer, starting replication: peer=server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.378Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: pipelining replication: peer="{Voter server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 07182d5d-d9ce-987d-e7fc-1076de26d2e7}" testing.go:149: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.389Z [WARN] server-5b0f090b-d63c-f47d-2644-d377258d5290: no known peers, aborting election testing.go:149: cluster: 2021-10-08T13:01:04.483Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.378607107 +0000 UTC m=+4.393289841 (1 nodes), 52.98797ms from start of poll, 54.19263ms from cluster start. Timeout at 2021-10-08 13:01:04.483707834 +0000 UTC m=+4.498390555, 105.100714ms after stability testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.483Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: updating configuration: command=AddStaging server-id=server-5b0f090b-d63c-f47d-2644-d377258d5290 server-addr=5b0f090b-d63c-f47d-2644-d377258d5290 servers="[{Suffrage:Voter ID:server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 Address:07182d5d-d9ce-987d-e7fc-1076de26d2e7} {Suffrage:Voter ID:server-c09f793d-0175-c5d5-9e56-821ff4cbb94b Address:c09f793d-0175-c5d5-9e56-821ff4cbb94b} {Suffrage:Voter ID:server-5b0f090b-d63c-f47d-2644-d377258d5290 Address:5b0f090b-d63c-f47d-2644-d377258d5290}]" testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: added peer, starting replication: peer=server-5b0f090b-d63c-f47d-2644-d377258d5290 testing.go:149: server-5b0f090b-d63c-f47d-2644-d377258d5290: 2021-10-08T13:01:04.484Z [WARN] server-5b0f090b-d63c-f47d-2644-d377258d5290: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [WARN] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: appendEntries rejected, sending older logs: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" next=1 testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.484Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: pipelining replication: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" testing.go:149: cluster: 2021-10-08T13:01:04.489Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:04.594Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.489382057 +0000 UTC m=+4.504064776 (1 nodes), 720ns from start of poll, 164.967565ms from cluster start. Timeout at 2021-10-08 13:01:04.594490251 +0000 UTC m=+4.609172971, 105.108195ms after stability testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.594Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: aborting pipeline replication: peer="{Voter server-07182d5d-d9ce-987d-e7fc-1076de26d2e7 07182d5d-d9ce-987d-e7fc-1076de26d2e7}" testing.go:149: server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: 2021-10-08T13:01:04.594Z [INFO] server-c09f793d-0175-c5d5-9e56-821ff4cbb94b: aborting pipeline replication: peer="{Voter server-5b0f090b-d63c-f47d-2644-d377258d5290 5b0f090b-d63c-f47d-2644-d377258d5290}" === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.595Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0de03a50-d39e-5807-fe27-e556166dcf99 Address:0de03a50-d39e-5807-fe27-e556166dcf99}]" testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.595Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering follower state: follower="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Follower]" leader= testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [WARN] server-0de03a50-d39e-5807-fe27-e556166dcf99: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering candidate state: node="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Candidate]" term=2 testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: election won: tally=1 testLoggerAdapter verbose: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering leader state: leader="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Leader]" --- PASS: TestRaft_SingleNode (0.06s) testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.595Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0de03a50-d39e-5807-fe27-e556166dcf99 Address:0de03a50-d39e-5807-fe27-e556166dcf99}]" testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.595Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering follower state: follower="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Follower]" leader= testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [WARN] server-0de03a50-d39e-5807-fe27-e556166dcf99: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering candidate state: node="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Candidate]" term=2 testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: election won: tally=1 testing.go:149: server-0de03a50-d39e-5807-fe27-e556166dcf99: 2021-10-08T13:01:04.655Z [INFO] server-0de03a50-d39e-5807-fe27-e556166dcf99: entering leader state: leader="Node at 0de03a50-d39e-5807-fe27-e556166dcf99 [Leader]" === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: 2021-10-08T13:01:04.657Z [INFO] server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.657Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testLoggerAdapter verbose: server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: 2021-10-08T13:01:04.657Z [INFO] server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: entering follower state: follower="Node at 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 [Follower]" leader= testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.657Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering follower state: follower="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Follower]" leader= testLoggerAdapter verbose: server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: 2021-10-08T13:01:04.657Z [INFO] server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.657Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: 2021-10-08T13:01:04.657Z [INFO] server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: entering follower state: follower="Node at 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 [Follower]" leader= testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [WARN] server-8865f411-3d88-5efd-f081-8f03117d5cf0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering candidate state: node="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Candidate]" term=2 testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: election won: tally=2 testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering leader state: leader="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Leader]" testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: added peer, starting replication: peer=server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: added peer, starting replication: peer=server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: pipelining replication: peer="{Voter server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360}" testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: pipelining replication: peer="{Voter server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.823Z [INFO] stable state for Follower reached at 2021-10-08 13:01:04.718717862 +0000 UTC m=+4.733400622 (2 nodes), 61.285302ms from start of poll, 61.738289ms from cluster start. Timeout at 2021-10-08 13:01:04.823812708 +0000 UTC m=+4.838495454, 105.094832ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.823Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.929Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.824012309 +0000 UTC m=+4.838695046 (1 nodes), 770ns from start of poll, 167.032713ms from cluster start. Timeout at 2021-10-08 13:01:04.929130229 +0000 UTC m=+4.943812978, 105.117932ms after stability testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.939Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: aborting pipeline replication: peer="{Voter server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}" testLoggerAdapter verbose: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.939Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: aborting pipeline replication: peer="{Voter server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360}" --- PASS: TestRaft_TripleNode (0.28s) testing.go:149: server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: 2021-10-08T13:01:04.657Z [INFO] server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.657Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testing.go:149: server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: 2021-10-08T13:01:04.657Z [INFO] server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360: entering follower state: follower="Node at 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 [Follower]" leader= testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.657Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering follower state: follower="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Follower]" leader= testing.go:149: server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: 2021-10-08T13:01:04.657Z [INFO] server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 Address:3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360} {Suffrage:Voter ID:server-8865f411-3d88-5efd-f081-8f03117d5cf0 Address:8865f411-3d88-5efd-f081-8f03117d5cf0} {Suffrage:Voter ID:server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 Address:738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}]" testing.go:149: cluster: 2021-10-08T13:01:04.657Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: 2021-10-08T13:01:04.657Z [INFO] server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78: entering follower state: follower="Node at 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 [Follower]" leader= testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [WARN] server-8865f411-3d88-5efd-f081-8f03117d5cf0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering candidate state: node="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Candidate]" term=2 testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: election won: tally=2 testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: entering leader state: leader="Node at 8865f411-3d88-5efd-f081-8f03117d5cf0 [Leader]" testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: added peer, starting replication: peer=server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: added peer, starting replication: peer=server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: pipelining replication: peer="{Voter server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360}" testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.718Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: pipelining replication: peer="{Voter server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}" testing.go:149: cluster: 2021-10-08T13:01:04.823Z [INFO] stable state for Follower reached at 2021-10-08 13:01:04.718717862 +0000 UTC m=+4.733400622 (2 nodes), 61.285302ms from start of poll, 61.738289ms from cluster start. Timeout at 2021-10-08 13:01:04.823812708 +0000 UTC m=+4.838495454, 105.094832ms after stability testing.go:149: cluster: 2021-10-08T13:01:04.823Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:04.929Z [INFO] stable state for Leader reached at 2021-10-08 13:01:04.824012309 +0000 UTC m=+4.838695046 (1 nodes), 770ns from start of poll, 167.032713ms from cluster start. Timeout at 2021-10-08 13:01:04.929130229 +0000 UTC m=+4.943812978, 105.117932ms after stability testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.939Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: aborting pipeline replication: peer="{Voter server-738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78 738b4bcc-5a47-c6d3-fb62-a0cba1c9ca78}" testing.go:149: server-8865f411-3d88-5efd-f081-8f03117d5cf0: 2021-10-08T13:01:04.939Z [INFO] server-8865f411-3d88-5efd-f081-8f03117d5cf0: aborting pipeline replication: peer="{Voter server-3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360 3d68bd8c-4d84-e8e5-cf3d-64aa9ddc9360}" === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:04.942Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:04.942Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:04.942Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:04.942Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering follower state: follower="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Follower]" leader= testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:04.942Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:04.942Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:04.942Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering follower state: follower="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Follower]" leader= testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.005Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.005Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=2 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: election won: tally=2 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering leader state: leader="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Leader]" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: added peer, starting replication: peer=server-1657509d-3ae9-7f33-af4a-7f6745f0da28 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: pipelining replication: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.111Z [INFO] stable state for Follower reached at 2021-10-08 13:01:05.006116843 +0000 UTC m=+5.020799565 (2 nodes), 63.627418ms from start of poll, 64.282871ms from cluster start. Timeout at 2021-10-08 13:01:05.111216147 +0000 UTC m=+5.125898866, 105.099301ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.111Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.216Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.111366976 +0000 UTC m=+5.126049692 (1 nodes), 777ns from start of poll, 169.532998ms from cluster start. Timeout at 2021-10-08 13:01:05.216439655 +0000 UTC m=+5.231122432, 105.07274ms after stability testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.228Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.229Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.232Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.233Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to pipeline appendEntries: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="append pipeline closed" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.233Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.235Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to pipeline appendEntries: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="append pipeline closed" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.235Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: aborting pipeline replication: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.242Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.243Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.245Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.248Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.262Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.263Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.264Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.265Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.274Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-6795ae26-d663-6610-e65c-57ee565e38d9 time=50.100845ms testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.279Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.280Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-1657509d-3ae9-7f33-af4a-7f6745f0da28 time=59.052727ms testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-6795ae26-d663-6610-e65c-57ee565e38d9 time=60.339114ms testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: heartbeat timeout reached, starting election: last-leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering candidate state: node="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Candidate]" term=3 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.308Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since we have a leader: from=6795ae26-d663-6610-e65c-57ee565e38d9 leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [ERROR] server-6795ae26-d663-6610-e65c-57ee565e38d9: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: heartbeat timeout reached, starting election: last-leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=3 testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.316Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=3 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: Election timeout reached, restarting election testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=4 testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.377Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering follower state: follower="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Follower]" leader= testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: election won: tally=2 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering leader state: leader="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Leader]" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.383Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.387Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.399Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.406Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=4 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.418Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.427Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to contact: server-id=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d time=50.245507ms testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.436Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.444Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.475Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to contact: server-id=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d time=97.752537ms testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=5 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.481Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.482Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.377582772 +0000 UTC m=+5.392265498 (1 nodes), 145.505221ms from start of poll, 435.748804ms from cluster start. Timeout at 2021-10-08 13:01:05.482655613 +0000 UTC m=+5.497338346, 105.072848ms after stability testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.534Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.535Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=6 testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.535Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.535Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: peer has newer term, stopping replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering follower state: follower="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Follower]" leader= testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.615Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.615Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=7 testLoggerAdapter verbose: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.615Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.615Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since our last term is greater: candidate=604ce575-9a61-d30c-c0d4-4bc15fb7c85d last-term=4 last-candidate-term=2 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=8 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: election won: tally=2 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering leader state: leader="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Leader]" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.623Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testLoggerAdapter verbose: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.623Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to get previous log: previous-index=5 last-index=3 error="log not found" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: appendEntries rejected, sending older logs: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" next=4 testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.628Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testLoggerAdapter verbose: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.628Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" --- PASS: TestRaft_LeaderFail (0.69s) testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:04.942Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:04.942Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:04.942Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:04.942Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering follower state: follower="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Follower]" leader= testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:04.942Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d Address:604ce575-9a61-d30c-c0d4-4bc15fb7c85d} {Suffrage:Voter ID:server-1657509d-3ae9-7f33-af4a-7f6745f0da28 Address:1657509d-3ae9-7f33-af4a-7f6745f0da28} {Suffrage:Voter ID:server-6795ae26-d663-6610-e65c-57ee565e38d9 Address:6795ae26-d663-6610-e65c-57ee565e38d9}]" testing.go:149: cluster: 2021-10-08T13:01:04.942Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:04.942Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering follower state: follower="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Follower]" leader= testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.005Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.005Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=2 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: election won: tally=2 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering leader state: leader="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Leader]" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: added peer, starting replication: peer=server-1657509d-3ae9-7f33-af4a-7f6745f0da28 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: pipelining replication: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.006Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: cluster: 2021-10-08T13:01:05.111Z [INFO] stable state for Follower reached at 2021-10-08 13:01:05.006116843 +0000 UTC m=+5.020799565 (2 nodes), 63.627418ms from start of poll, 64.282871ms from cluster start. Timeout at 2021-10-08 13:01:05.111216147 +0000 UTC m=+5.125898866, 105.099301ms after stability testing.go:149: cluster: 2021-10-08T13:01:05.111Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:05.216Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.111366976 +0000 UTC m=+5.126049692 (1 nodes), 777ns from start of poll, 169.532998ms from cluster start. Timeout at 2021-10-08 13:01:05.216439655 +0000 UTC m=+5.231122432, 105.07274ms after stability raft_test.go:336: [INFO] Disconnecting Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Leader] testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.228Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.229Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: cluster: 2021-10-08T13:01:05.232Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.233Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to pipeline appendEntries: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="append pipeline closed" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.233Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.235Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to pipeline appendEntries: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="append pipeline closed" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.235Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: aborting pipeline replication: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.242Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.243Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.245Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.248Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.262Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.263Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.264Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=6795ae26-d663-6610-e65c-57ee565e38d9 error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.265Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to heartbeat to: peer=1657509d-3ae9-7f33-af4a-7f6745f0da28 error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.274Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-6795ae26-d663-6610-e65c-57ee565e38d9 time=50.100845ms testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.279Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.280Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to appendEntries to: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-1657509d-3ae9-7f33-af4a-7f6745f0da28 time=59.052727ms testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact: server-id=server-6795ae26-d663-6610-e65c-57ee565e38d9 time=60.339114ms testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to contact quorum of nodes, stepping down testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.284Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: heartbeat timeout reached, starting election: last-leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering candidate state: node="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Candidate]" term=3 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.308Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since we have a leader: from=6795ae26-d663-6610-e65c-57ee565e38d9 leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.308Z [ERROR] server-6795ae26-d663-6610-e65c-57ee565e38d9: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: heartbeat timeout reached, starting election: last-leader=604ce575-9a61-d30c-c0d4-4bc15fb7c85d testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=3 testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.316Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: duplicate requestVote for same term: term=3 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.316Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=3 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.342Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: Election timeout reached, restarting election testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=4 testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.377Z [INFO] server-6795ae26-d663-6610-e65c-57ee565e38d9: entering follower state: follower="Node at 6795ae26-d663-6610-e65c-57ee565e38d9 [Follower]" leader= testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: election won: tally=2 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering leader state: leader="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Leader]" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to make requestVote RPC: target="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.377Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.383Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.387Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.399Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.406Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=4 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.406Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.418Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.427Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to contact: server-id=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d time=50.245507ms testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.436Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.444Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to heartbeat to: peer=604ce575-9a61-d30c-c0d4-4bc15fb7c85d error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.475Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to contact: server-id=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d time=97.752537ms testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=5 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" error="failed to connect to peer: 6795ae26-d663-6610-e65c-57ee565e38d9" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.480Z [ERROR] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to make requestVote RPC: target="{Voter server-1657509d-3ae9-7f33-af4a-7f6745f0da28 1657509d-3ae9-7f33-af4a-7f6745f0da28}" error="failed to connect to peer: 1657509d-3ae9-7f33-af4a-7f6745f0da28" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.481Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: failed to appendEntries to: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" error="failed to connect to peer: 604ce575-9a61-d30c-c0d4-4bc15fb7c85d" testing.go:149: cluster: 2021-10-08T13:01:05.482Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.377582772 +0000 UTC m=+5.392265498 (1 nodes), 145.505221ms from start of poll, 435.748804ms from cluster start. Timeout at 2021-10-08 13:01:05.482655613 +0000 UTC m=+5.497338346, 105.072848ms after stability raft_test.go:371: [INFO] Reconnecting Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate] testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.534Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.535Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=6 testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.535Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.535Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [ERROR] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: peer has newer term, stopping replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering follower state: follower="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Follower]" leader= testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.563Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.615Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: Election timeout reached, restarting election testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.615Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering candidate state: node="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Candidate]" term=7 testing.go:149: server-6795ae26-d663-6610-e65c-57ee565e38d9: 2021-10-08T13:01:05.615Z [WARN] server-6795ae26-d663-6610-e65c-57ee565e38d9: rejecting vote request since we have a leader: from=604ce575-9a61-d30c-c0d4-4bc15fb7c85d leader=1657509d-3ae9-7f33-af4a-7f6745f0da28 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.615Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: rejecting vote request since our last term is greater: candidate=604ce575-9a61-d30c-c0d4-4bc15fb7c85d last-term=4 last-candidate-term=2 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering candidate state: node="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Candidate]" term=8 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: election won: tally=2 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: entering leader state: leader="Node at 1657509d-3ae9-7f33-af4a-7f6745f0da28 [Leader]" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.623Z [INFO] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: entering follower state: follower="Node at 604ce575-9a61-d30c-c0d4-4bc15fb7c85d [Follower]" leader= testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: added peer, starting replication: peer=server-6795ae26-d663-6610-e65c-57ee565e38d9 testing.go:149: server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: 2021-10-08T13:01:05.623Z [WARN] server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d: failed to get previous log: previous-index=5 last-index=3 error="log not found" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [WARN] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: appendEntries rejected, sending older logs: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" next=4 testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.623Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: pipelining replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.628Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-6795ae26-d663-6610-e65c-57ee565e38d9 6795ae26-d663-6610-e65c-57ee565e38d9}" testing.go:149: server-1657509d-3ae9-7f33-af4a-7f6745f0da28: 2021-10-08T13:01:05.628Z [INFO] server-1657509d-3ae9-7f33-af4a-7f6745f0da28: aborting pipeline replication: peer="{Voter server-604ce575-9a61-d30c-c0d4-4bc15fb7c85d 604ce575-9a61-d30c-c0d4-4bc15fb7c85d}" === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: 2021-10-08T13:01:05.630Z [INFO] server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testLoggerAdapter verbose: server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: 2021-10-08T13:01:05.630Z [INFO] server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: entering follower state: follower="Node at cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 [Follower]" leader= testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.630Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.631Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering follower state: follower="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Follower]" leader= testLoggerAdapter verbose: server-d3ca39a5-e833-ea14-5474-bc04b4db8808: 2021-10-08T13:01:05.631Z [INFO] server-d3ca39a5-e833-ea14-5474-bc04b4db8808: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.631Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d3ca39a5-e833-ea14-5474-bc04b4db8808: 2021-10-08T13:01:05.631Z [INFO] server-d3ca39a5-e833-ea14-5474-bc04b4db8808: entering follower state: follower="Node at d3ca39a5-e833-ea14-5474-bc04b4db8808 [Follower]" leader= testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [WARN] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering candidate state: node="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Candidate]" term=2 testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: election won: tally=2 testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering leader state: leader="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Leader]" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: added peer, starting replication: peer=server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: added peer, starting replication: peer=server-d3ca39a5-e833-ea14-5474-bc04b4db8808 testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-d3ca39a5-e833-ea14-5474-bc04b4db8808 d3ca39a5-e833-ea14-5474-bc04b4db8808}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.798Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.69353978 +0000 UTC m=+5.708222510 (1 nodes), 62.32334ms from start of poll, 62.942029ms from cluster start. Timeout at 2021-10-08 13:01:05.798639266 +0000 UTC m=+5.813321990, 105.09948ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.798Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.903Z [INFO] stable state for Follower reached at 2021-10-08 13:01:05.798803545 +0000 UTC m=+5.813486292 (2 nodes), 1.437µs from start of poll, 168.205811ms from cluster start. Timeout at 2021-10-08 13:01:05.903868 +0000 UTC m=+5.918550742, 105.06445ms after stability testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [ERROR] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: failed to pipeline appendEntries: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" error="append pipeline closed" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [ERROR] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: failed to appendEntries to: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" error="failed to connect to peer: cbca2536-f920-dbc5-1b7f-62b7d8fa85c9" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.914Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:05.926Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.031Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.926221815 +0000 UTC m=+5.940904564 (1 nodes), 1.095µs from start of poll, 295.624083ms from cluster start. Timeout at 2021-10-08 13:01:06.031340213 +0000 UTC m=+6.046022957, 105.118393ms after stability testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:06.031Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-d3ca39a5-e833-ea14-5474-bc04b4db8808 d3ca39a5-e833-ea14-5474-bc04b4db8808}" testLoggerAdapter verbose: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:06.031Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" --- PASS: TestRaft_BehindFollower (0.40s) testing.go:149: server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: 2021-10-08T13:01:05.630Z [INFO] server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testing.go:149: server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: 2021-10-08T13:01:05.630Z [INFO] server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9: entering follower state: follower="Node at cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 [Follower]" leader= testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.630Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.631Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering follower state: follower="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Follower]" leader= testing.go:149: server-d3ca39a5-e833-ea14-5474-bc04b4db8808: 2021-10-08T13:01:05.631Z [INFO] server-d3ca39a5-e833-ea14-5474-bc04b4db8808: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 Address:cbca2536-f920-dbc5-1b7f-62b7d8fa85c9} {Suffrage:Voter ID:server-deea8aa2-8230-02ee-56b6-6be4c59152a3 Address:deea8aa2-8230-02ee-56b6-6be4c59152a3} {Suffrage:Voter ID:server-d3ca39a5-e833-ea14-5474-bc04b4db8808 Address:d3ca39a5-e833-ea14-5474-bc04b4db8808}]" testing.go:149: cluster: 2021-10-08T13:01:05.631Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-d3ca39a5-e833-ea14-5474-bc04b4db8808: 2021-10-08T13:01:05.631Z [INFO] server-d3ca39a5-e833-ea14-5474-bc04b4db8808: entering follower state: follower="Node at d3ca39a5-e833-ea14-5474-bc04b4db8808 [Follower]" leader= testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [WARN] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering candidate state: node="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Candidate]" term=2 testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: election won: tally=2 testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: entering leader state: leader="Node at deea8aa2-8230-02ee-56b6-6be4c59152a3 [Leader]" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: added peer, starting replication: peer=server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: added peer, starting replication: peer=server-d3ca39a5-e833-ea14-5474-bc04b4db8808 testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.693Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-d3ca39a5-e833-ea14-5474-bc04b4db8808 d3ca39a5-e833-ea14-5474-bc04b4db8808}" testing.go:149: cluster: 2021-10-08T13:01:05.798Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.69353978 +0000 UTC m=+5.708222510 (1 nodes), 62.32334ms from start of poll, 62.942029ms from cluster start. Timeout at 2021-10-08 13:01:05.798639266 +0000 UTC m=+5.813321990, 105.09948ms after stability testing.go:149: cluster: 2021-10-08T13:01:05.798Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:05.903Z [INFO] stable state for Follower reached at 2021-10-08 13:01:05.798803545 +0000 UTC m=+5.813486292 (2 nodes), 1.437µs from start of poll, 168.205811ms from cluster start. Timeout at 2021-10-08 13:01:05.903868 +0000 UTC m=+5.918550742, 105.06445ms after stability testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [ERROR] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: failed to pipeline appendEntries: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" error="append pipeline closed" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.904Z [ERROR] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: failed to appendEntries to: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" error="failed to connect to peer: cbca2536-f920-dbc5-1b7f-62b7d8fa85c9" raft_test.go:420: [INFO] Finished apply without behind follower testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:05.914Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: pipelining replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" testing.go:149: cluster: 2021-10-08T13:01:05.926Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:06.031Z [INFO] stable state for Leader reached at 2021-10-08 13:01:05.926221815 +0000 UTC m=+5.940904564 (1 nodes), 1.095µs from start of poll, 295.624083ms from cluster start. Timeout at 2021-10-08 13:01:06.031340213 +0000 UTC m=+6.046022957, 105.118393ms after stability testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:06.031Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-d3ca39a5-e833-ea14-5474-bc04b4db8808 d3ca39a5-e833-ea14-5474-bc04b4db8808}" testing.go:149: server-deea8aa2-8230-02ee-56b6-6be4c59152a3: 2021-10-08T13:01:06.031Z [INFO] server-deea8aa2-8230-02ee-56b6-6be4c59152a3: aborting pipeline replication: peer="{Voter server-cbca2536-f920-dbc5-1b7f-62b7d8fa85c9 cbca2536-f920-dbc5-1b7f-62b7d8fa85c9}" === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-6084681b-5a00-313b-9ac8-d4135029b712: 2021-10-08T13:01:06.034Z [INFO] server-6084681b-5a00-313b-9ac8-d4135029b712: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.034Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testLoggerAdapter verbose: server-6084681b-5a00-313b-9ac8-d4135029b712: 2021-10-08T13:01:06.034Z [INFO] server-6084681b-5a00-313b-9ac8-d4135029b712: entering follower state: follower="Node at 6084681b-5a00-313b-9ac8-d4135029b712 [Follower]" leader= testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.034Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering follower state: follower="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Follower]" leader= testLoggerAdapter verbose: server-67e01f0f-3b35-2914-e63d-1262751cbe6f: 2021-10-08T13:01:06.035Z [INFO] server-67e01f0f-3b35-2914-e63d-1262751cbe6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.035Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-67e01f0f-3b35-2914-e63d-1262751cbe6f: 2021-10-08T13:01:06.035Z [INFO] server-67e01f0f-3b35-2914-e63d-1262751cbe6f: entering follower state: follower="Node at 67e01f0f-3b35-2914-e63d-1262751cbe6f [Follower]" leader= testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.085Z [WARN] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering candidate state: node="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Candidate]" term=2 testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: election won: tally=2 testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering leader state: leader="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Leader]" testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: added peer, starting replication: peer=server-6084681b-5a00-313b-9ac8-d4135029b712 testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: added peer, starting replication: peer=server-67e01f0f-3b35-2914-e63d-1262751cbe6f testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: pipelining replication: peer="{Voter server-6084681b-5a00-313b-9ac8-d4135029b712 6084681b-5a00-313b-9ac8-d4135029b712}" testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: pipelining replication: peer="{Voter server-67e01f0f-3b35-2914-e63d-1262751cbe6f 67e01f0f-3b35-2914-e63d-1262751cbe6f}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.191Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.086278343 +0000 UTC m=+6.100961110 (1 nodes), 51.169978ms from start of poll, 51.898747ms from cluster start. Timeout at 2021-10-08 13:01:06.191380157 +0000 UTC m=+6.206062891, 105.101781ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.191Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.296Z [INFO] stable state for Follower reached at 2021-10-08 13:01:06.191525898 +0000 UTC m=+6.206208614 (2 nodes), 911ns from start of poll, 157.146251ms from cluster start. Timeout at 2021-10-08 13:01:06.296635901 +0000 UTC m=+6.311318652, 105.110038ms after stability testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.296Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: aborting pipeline replication: peer="{Voter server-67e01f0f-3b35-2914-e63d-1262751cbe6f 67e01f0f-3b35-2914-e63d-1262751cbe6f}" testLoggerAdapter verbose: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.296Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: aborting pipeline replication: peer="{Voter server-6084681b-5a00-313b-9ac8-d4135029b712 6084681b-5a00-313b-9ac8-d4135029b712}" --- PASS: TestRaft_ApplyNonLeader (0.26s) testing.go:149: server-6084681b-5a00-313b-9ac8-d4135029b712: 2021-10-08T13:01:06.034Z [INFO] server-6084681b-5a00-313b-9ac8-d4135029b712: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.034Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testing.go:149: server-6084681b-5a00-313b-9ac8-d4135029b712: 2021-10-08T13:01:06.034Z [INFO] server-6084681b-5a00-313b-9ac8-d4135029b712: entering follower state: follower="Node at 6084681b-5a00-313b-9ac8-d4135029b712 [Follower]" leader= testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.034Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering follower state: follower="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Follower]" leader= testing.go:149: server-67e01f0f-3b35-2914-e63d-1262751cbe6f: 2021-10-08T13:01:06.035Z [INFO] server-67e01f0f-3b35-2914-e63d-1262751cbe6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6084681b-5a00-313b-9ac8-d4135029b712 Address:6084681b-5a00-313b-9ac8-d4135029b712} {Suffrage:Voter ID:server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f Address:d4cef6ff-3415-eb7a-9791-d7a0cb249a0f} {Suffrage:Voter ID:server-67e01f0f-3b35-2914-e63d-1262751cbe6f Address:67e01f0f-3b35-2914-e63d-1262751cbe6f}]" testing.go:149: cluster: 2021-10-08T13:01:06.035Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-67e01f0f-3b35-2914-e63d-1262751cbe6f: 2021-10-08T13:01:06.035Z [INFO] server-67e01f0f-3b35-2914-e63d-1262751cbe6f: entering follower state: follower="Node at 67e01f0f-3b35-2914-e63d-1262751cbe6f [Follower]" leader= testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.085Z [WARN] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering candidate state: node="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Candidate]" term=2 testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: election won: tally=2 testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: entering leader state: leader="Node at d4cef6ff-3415-eb7a-9791-d7a0cb249a0f [Leader]" testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: added peer, starting replication: peer=server-6084681b-5a00-313b-9ac8-d4135029b712 testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: added peer, starting replication: peer=server-67e01f0f-3b35-2914-e63d-1262751cbe6f testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: pipelining replication: peer="{Voter server-6084681b-5a00-313b-9ac8-d4135029b712 6084681b-5a00-313b-9ac8-d4135029b712}" testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.086Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: pipelining replication: peer="{Voter server-67e01f0f-3b35-2914-e63d-1262751cbe6f 67e01f0f-3b35-2914-e63d-1262751cbe6f}" testing.go:149: cluster: 2021-10-08T13:01:06.191Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.086278343 +0000 UTC m=+6.100961110 (1 nodes), 51.169978ms from start of poll, 51.898747ms from cluster start. Timeout at 2021-10-08 13:01:06.191380157 +0000 UTC m=+6.206062891, 105.101781ms after stability testing.go:149: cluster: 2021-10-08T13:01:06.191Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:06.296Z [INFO] stable state for Follower reached at 2021-10-08 13:01:06.191525898 +0000 UTC m=+6.206208614 (2 nodes), 911ns from start of poll, 157.146251ms from cluster start. Timeout at 2021-10-08 13:01:06.296635901 +0000 UTC m=+6.311318652, 105.110038ms after stability testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.296Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: aborting pipeline replication: peer="{Voter server-67e01f0f-3b35-2914-e63d-1262751cbe6f 67e01f0f-3b35-2914-e63d-1262751cbe6f}" testing.go:149: server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: 2021-10-08T13:01:06.296Z [INFO] server-d4cef6ff-3415-eb7a-9791-d7a0cb249a0f: aborting pipeline replication: peer="{Voter server-6084681b-5a00-313b-9ac8-d4135029b712 6084681b-5a00-313b-9ac8-d4135029b712}" === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-e3e12707-0293-78a3-bf3b-00caff824e88: 2021-10-08T13:01:06.298Z [INFO] server-e3e12707-0293-78a3-bf3b-00caff824e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testLoggerAdapter verbose: server-e3e12707-0293-78a3-bf3b-00caff824e88: 2021-10-08T13:01:06.298Z [INFO] server-e3e12707-0293-78a3-bf3b-00caff824e88: entering follower state: follower="Node at e3e12707-0293-78a3-bf3b-00caff824e88 [Follower]" leader= testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.298Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.298Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering follower state: follower="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Follower]" leader= testLoggerAdapter verbose: server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: 2021-10-08T13:01:06.298Z [INFO] server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.298Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: 2021-10-08T13:01:06.298Z [INFO] server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: entering follower state: follower="Node at 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea [Follower]" leader= testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [WARN] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering candidate state: node="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Candidate]" term=2 testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: election won: tally=2 testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering leader state: leader="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Leader]" testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: added peer, starting replication: peer=server-e3e12707-0293-78a3-bf3b-00caff824e88 testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: added peer, starting replication: peer=server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: pipelining replication: peer="{Voter server-e3e12707-0293-78a3-bf3b-00caff824e88 e3e12707-0293-78a3-bf3b-00caff824e88}" testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: pipelining replication: peer="{Voter server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.639Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.434644569 +0000 UTC m=+6.449327308 (1 nodes), 136.081092ms from start of poll, 136.4995ms from cluster start. Timeout at 2021-10-08 13:01:06.639730768 +0000 UTC m=+6.654413507, 205.086199ms after stability testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.651Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: aborting pipeline replication: peer="{Voter server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}" testLoggerAdapter verbose: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.651Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: aborting pipeline replication: peer="{Voter server-e3e12707-0293-78a3-bf3b-00caff824e88 e3e12707-0293-78a3-bf3b-00caff824e88}" --- PASS: TestRaft_ApplyConcurrent (0.36s) testing.go:149: server-e3e12707-0293-78a3-bf3b-00caff824e88: 2021-10-08T13:01:06.298Z [INFO] server-e3e12707-0293-78a3-bf3b-00caff824e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testing.go:149: server-e3e12707-0293-78a3-bf3b-00caff824e88: 2021-10-08T13:01:06.298Z [INFO] server-e3e12707-0293-78a3-bf3b-00caff824e88: entering follower state: follower="Node at e3e12707-0293-78a3-bf3b-00caff824e88 [Follower]" leader= testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.298Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.298Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering follower state: follower="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Follower]" leader= testing.go:149: server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: 2021-10-08T13:01:06.298Z [INFO] server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e3e12707-0293-78a3-bf3b-00caff824e88 Address:e3e12707-0293-78a3-bf3b-00caff824e88} {Suffrage:Voter ID:server-753d021f-1c74-2f9e-7e82-dbbe0d12af93 Address:753d021f-1c74-2f9e-7e82-dbbe0d12af93} {Suffrage:Voter ID:server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea Address:2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}]" testing.go:149: cluster: 2021-10-08T13:01:06.298Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: 2021-10-08T13:01:06.298Z [INFO] server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea: entering follower state: follower="Node at 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea [Follower]" leader= testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [WARN] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering candidate state: node="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Candidate]" term=2 testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: election won: tally=2 testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: entering leader state: leader="Node at 753d021f-1c74-2f9e-7e82-dbbe0d12af93 [Leader]" testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: added peer, starting replication: peer=server-e3e12707-0293-78a3-bf3b-00caff824e88 testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: added peer, starting replication: peer=server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: pipelining replication: peer="{Voter server-e3e12707-0293-78a3-bf3b-00caff824e88 e3e12707-0293-78a3-bf3b-00caff824e88}" testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.434Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: pipelining replication: peer="{Voter server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}" testing.go:149: cluster: 2021-10-08T13:01:06.639Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.434644569 +0000 UTC m=+6.449327308 (1 nodes), 136.081092ms from start of poll, 136.4995ms from cluster start. Timeout at 2021-10-08 13:01:06.639730768 +0000 UTC m=+6.654413507, 205.086199ms after stability testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.651Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: aborting pipeline replication: peer="{Voter server-2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea 2d67c2de-658c-3c3b-e6bf-4bd5bc4d47ea}" testing.go:149: server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: 2021-10-08T13:01:06.651Z [INFO] server-753d021f-1c74-2f9e-7e82-dbbe0d12af93: aborting pipeline replication: peer="{Voter server-e3e12707-0293-78a3-bf3b-00caff824e88 e3e12707-0293-78a3-bf3b-00caff824e88}" === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.653Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 Address:a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46}]" testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.653Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering follower state: follower="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.653Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [WARN] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering candidate state: node="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Candidate]" term=2 testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: election won: tally=1 testLoggerAdapter verbose: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering leader state: leader="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:06.998Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.797554592 +0000 UTC m=+6.812237317 (1 nodes), 143.875389ms from start of poll, 144.591361ms from cluster start. Timeout at 2021-10-08 13:01:06.998658923 +0000 UTC m=+7.013341647, 201.10433ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.35s) testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.653Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 Address:a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46}]" testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.653Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering follower state: follower="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:06.653Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [WARN] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering candidate state: node="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Candidate]" term=2 testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: election won: tally=1 testing.go:149: server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: 2021-10-08T13:01:06.797Z [INFO] server-a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46: entering leader state: leader="Node at a6a28d37-b0c0-5d3a-6dd3-4f65f9742b46 [Leader]" testing.go:149: cluster: 2021-10-08T13:01:06.998Z [INFO] stable state for Leader reached at 2021-10-08 13:01:06.797554592 +0000 UTC m=+6.812237317 (1 nodes), 143.875389ms from start of poll, 144.591361ms from cluster start. Timeout at 2021-10-08 13:01:06.998658923 +0000 UTC m=+7.013341647, 201.10433ms after stability === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.000Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600}]" testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.000Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering follower state: follower="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Follower]" leader= testLoggerAdapter verbose: server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: 2021-10-08T13:01:07.000Z [INFO] server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600}]" testLoggerAdapter verbose: server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: 2021-10-08T13:01:07.000Z [INFO] server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: entering follower state: follower="Node at b7a65e88-e93a-e347-fae3-0ed6d35c7600 [Follower]" leader= testLoggerAdapter verbose: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.000Z [INFO] server-3155d841-3580-75a9-808c-0eae60e20c03: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.000Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.000Z [INFO] server-3155d841-3580-75a9-808c-0eae60e20c03: entering follower state: follower="Node at 3155d841-3580-75a9-808c-0eae60e20c03 [Follower]" leader= testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [WARN] server-92efc6e9-b15b-08ea-2710-afca4cd34386: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering candidate state: node="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Candidate]" term=2 testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: election won: tally=2 testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering leader state: leader="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Leader]" testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: added peer, starting replication: peer=server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: pipelining replication: peer="{Voter server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 b7a65e88-e93a-e347-fae3-0ed6d35c7600}" testLoggerAdapter verbose: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.065Z [WARN] server-3155d841-3580-75a9-808c-0eae60e20c03: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.165Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.060518931 +0000 UTC m=+7.075201683 (1 nodes), 59.699195ms from start of poll, 60.531694ms from cluster start. Timeout at 2021-10-08 13:01:07.165623012 +0000 UTC m=+7.180305752, 105.104069ms after stability testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.165Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: updating configuration: command=AddStaging server-id=server-3155d841-3580-75a9-808c-0eae60e20c03 server-addr=3155d841-3580-75a9-808c-0eae60e20c03 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600} {Suffrage:Voter ID:server-3155d841-3580-75a9-808c-0eae60e20c03 Address:3155d841-3580-75a9-808c-0eae60e20c03}]" testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.165Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: added peer, starting replication: peer=server-3155d841-3580-75a9-808c-0eae60e20c03 testLoggerAdapter verbose: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.166Z [WARN] server-3155d841-3580-75a9-808c-0eae60e20c03: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.166Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.166Z [WARN] server-92efc6e9-b15b-08ea-2710-afca4cd34386: appendEntries rejected, sending older logs: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" next=1 testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.166Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: pipelining replication: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.271Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.166232154 +0000 UTC m=+7.180914888 (1 nodes), 917ns from start of poll, 166.244899ms from cluster start. Timeout at 2021-10-08 13:01:07.271359402 +0000 UTC m=+7.286042147, 105.127259ms after stability testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.271Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: aborting pipeline replication: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" testLoggerAdapter verbose: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.271Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: aborting pipeline replication: peer="{Voter server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 b7a65e88-e93a-e347-fae3-0ed6d35c7600}" --- PASS: TestRaft_JoinNode (0.27s) testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.000Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600}]" testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.000Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering follower state: follower="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Follower]" leader= testing.go:149: server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: 2021-10-08T13:01:07.000Z [INFO] server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600}]" testing.go:149: server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: 2021-10-08T13:01:07.000Z [INFO] server-b7a65e88-e93a-e347-fae3-0ed6d35c7600: entering follower state: follower="Node at b7a65e88-e93a-e347-fae3-0ed6d35c7600 [Follower]" leader= testing.go:149: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.000Z [INFO] server-3155d841-3580-75a9-808c-0eae60e20c03: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-08T13:01:07.000Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.000Z [INFO] server-3155d841-3580-75a9-808c-0eae60e20c03: entering follower state: follower="Node at 3155d841-3580-75a9-808c-0eae60e20c03 [Follower]" leader= testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [WARN] server-92efc6e9-b15b-08ea-2710-afca4cd34386: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering candidate state: node="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Candidate]" term=2 testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: election won: tally=2 testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: entering leader state: leader="Node at 92efc6e9-b15b-08ea-2710-afca4cd34386 [Leader]" testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: added peer, starting replication: peer=server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.060Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: pipelining replication: peer="{Voter server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 b7a65e88-e93a-e347-fae3-0ed6d35c7600}" testing.go:149: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.065Z [WARN] server-3155d841-3580-75a9-808c-0eae60e20c03: no known peers, aborting election testing.go:149: cluster: 2021-10-08T13:01:07.165Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.060518931 +0000 UTC m=+7.075201683 (1 nodes), 59.699195ms from start of poll, 60.531694ms from cluster start. Timeout at 2021-10-08 13:01:07.165623012 +0000 UTC m=+7.180305752, 105.104069ms after stability testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.165Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: updating configuration: command=AddStaging server-id=server-3155d841-3580-75a9-808c-0eae60e20c03 server-addr=3155d841-3580-75a9-808c-0eae60e20c03 servers="[{Suffrage:Voter ID:server-92efc6e9-b15b-08ea-2710-afca4cd34386 Address:92efc6e9-b15b-08ea-2710-afca4cd34386} {Suffrage:Voter ID:server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 Address:b7a65e88-e93a-e347-fae3-0ed6d35c7600} {Suffrage:Voter ID:server-3155d841-3580-75a9-808c-0eae60e20c03 Address:3155d841-3580-75a9-808c-0eae60e20c03}]" testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.165Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: added peer, starting replication: peer=server-3155d841-3580-75a9-808c-0eae60e20c03 testing.go:149: server-3155d841-3580-75a9-808c-0eae60e20c03: 2021-10-08T13:01:07.166Z [WARN] server-3155d841-3580-75a9-808c-0eae60e20c03: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: cluster: 2021-10-08T13:01:07.166Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.166Z [WARN] server-92efc6e9-b15b-08ea-2710-afca4cd34386: appendEntries rejected, sending older logs: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" next=1 testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.166Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: pipelining replication: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" testing.go:149: cluster: 2021-10-08T13:01:07.271Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.166232154 +0000 UTC m=+7.180914888 (1 nodes), 917ns from start of poll, 166.244899ms from cluster start. Timeout at 2021-10-08 13:01:07.271359402 +0000 UTC m=+7.286042147, 105.127259ms after stability testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.271Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: aborting pipeline replication: peer="{Voter server-3155d841-3580-75a9-808c-0eae60e20c03 3155d841-3580-75a9-808c-0eae60e20c03}" testing.go:149: server-92efc6e9-b15b-08ea-2710-afca4cd34386: 2021-10-08T13:01:07.271Z [INFO] server-92efc6e9-b15b-08ea-2710-afca4cd34386: aborting pipeline replication: peer="{Voter server-b7a65e88-e93a-e347-fae3-0ed6d35c7600 b7a65e88-e93a-e347-fae3-0ed6d35c7600}" === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.273Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb}]" testLoggerAdapter verbose: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.273Z [INFO] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.274Z [INFO] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.274Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.274Z [INFO] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: entering follower state: follower="Node at e304b84e-17af-a6a8-ff5a-afd243d28d08 [Follower]" leader= testLoggerAdapter verbose: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.274Z [INFO] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: entering follower state: follower="Node at 737be045-7aa9-8b68-fc51-d9cea4d8cae1 [Follower]" leader= testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.274Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering follower state: follower="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Follower]" leader= testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering candidate state: node="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Candidate]" term=2 testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: election won: tally=1 testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering leader state: leader="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Leader]" testLoggerAdapter verbose: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.361Z [WARN] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: no known peers, aborting election testLoggerAdapter verbose: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.371Z [WARN] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.443Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.338570245 +0000 UTC m=+7.353252995 (1 nodes), 64.316873ms from start of poll, 65.600445ms from cluster start. Timeout at 2021-10-08 13:01:07.443690166 +0000 UTC m=+7.458372911, 105.119916ms after stability testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.443Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: updating configuration: command=AddStaging server-id=server-e304b84e-17af-a6a8-ff5a-afd243d28d08 server-addr=e304b84e-17af-a6a8-ff5a-afd243d28d08 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb} {Suffrage:Voter ID:server-e304b84e-17af-a6a8-ff5a-afd243d28d08 Address:e304b84e-17af-a6a8-ff5a-afd243d28d08}]" testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.443Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: added peer, starting replication: peer=server-e304b84e-17af-a6a8-ff5a-afd243d28d08 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.444Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.444Z [WARN] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.444Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: appendEntries rejected, sending older logs: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" next=1 testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.444Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: pipelining replication: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.549Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.444106905 +0000 UTC m=+7.458789632 (1 nodes), 807ns from start of poll, 171.137082ms from cluster start. Timeout at 2021-10-08 13:01:07.5491945 +0000 UTC m=+7.563877240, 105.087608ms after stability testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: updating configuration: command=AddStaging server-id=server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 server-addr=737be045-7aa9-8b68-fc51-d9cea4d8cae1 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb} {Suffrage:Voter ID:server-e304b84e-17af-a6a8-ff5a-afd243d28d08 Address:e304b84e-17af-a6a8-ff5a-afd243d28d08} {Suffrage:Voter ID:server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 Address:737be045-7aa9-8b68-fc51-d9cea4d8cae1}]" testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: added peer, starting replication: peer=server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 testLoggerAdapter verbose: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.549Z [WARN] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: appendEntries rejected, sending older logs: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" next=1 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.549Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: pipelining replication: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.655Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.549771164 +0000 UTC m=+7.564453899 (1 nodes), 907ns from start of poll, 276.801349ms from cluster start. Timeout at 2021-10-08 13:01:07.654896368 +0000 UTC m=+7.669579113, 105.125214ms after stability testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.655Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: aborting pipeline replication: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" testLoggerAdapter verbose: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.655Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: aborting pipeline replication: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" --- PASS: TestRaft_JoinNode_ConfigStore (0.38s) testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.273Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb}]" testing.go:149: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.273Z [INFO] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: initial configuration: index=0 servers=[] testing.go:149: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.274Z [INFO] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-08T13:01:07.274Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.274Z [INFO] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: entering follower state: follower="Node at e304b84e-17af-a6a8-ff5a-afd243d28d08 [Follower]" leader= testing.go:149: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.274Z [INFO] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: entering follower state: follower="Node at 737be045-7aa9-8b68-fc51-d9cea4d8cae1 [Follower]" leader= testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.274Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering follower state: follower="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Follower]" leader= testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering candidate state: node="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Candidate]" term=2 testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: election won: tally=1 testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.338Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: entering leader state: leader="Node at 2e582ec0-4b59-10e5-4429-3788f5f9ceeb [Leader]" testing.go:149: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.361Z [WARN] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: no known peers, aborting election testing.go:149: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.371Z [WARN] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: no known peers, aborting election testing.go:149: cluster: 2021-10-08T13:01:07.443Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.338570245 +0000 UTC m=+7.353252995 (1 nodes), 64.316873ms from start of poll, 65.600445ms from cluster start. Timeout at 2021-10-08 13:01:07.443690166 +0000 UTC m=+7.458372911, 105.119916ms after stability testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.443Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: updating configuration: command=AddStaging server-id=server-e304b84e-17af-a6a8-ff5a-afd243d28d08 server-addr=e304b84e-17af-a6a8-ff5a-afd243d28d08 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb} {Suffrage:Voter ID:server-e304b84e-17af-a6a8-ff5a-afd243d28d08 Address:e304b84e-17af-a6a8-ff5a-afd243d28d08}]" testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.443Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: added peer, starting replication: peer=server-e304b84e-17af-a6a8-ff5a-afd243d28d08 testing.go:149: cluster: 2021-10-08T13:01:07.444Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e304b84e-17af-a6a8-ff5a-afd243d28d08: 2021-10-08T13:01:07.444Z [WARN] server-e304b84e-17af-a6a8-ff5a-afd243d28d08: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.444Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: appendEntries rejected, sending older logs: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" next=1 testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.444Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: pipelining replication: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" testing.go:149: cluster: 2021-10-08T13:01:07.549Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.444106905 +0000 UTC m=+7.458789632 (1 nodes), 807ns from start of poll, 171.137082ms from cluster start. Timeout at 2021-10-08 13:01:07.5491945 +0000 UTC m=+7.563877240, 105.087608ms after stability testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: updating configuration: command=AddStaging server-id=server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 server-addr=737be045-7aa9-8b68-fc51-d9cea4d8cae1 servers="[{Suffrage:Voter ID:server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb Address:2e582ec0-4b59-10e5-4429-3788f5f9ceeb} {Suffrage:Voter ID:server-e304b84e-17af-a6a8-ff5a-afd243d28d08 Address:e304b84e-17af-a6a8-ff5a-afd243d28d08} {Suffrage:Voter ID:server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 Address:737be045-7aa9-8b68-fc51-d9cea4d8cae1}]" testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: added peer, starting replication: peer=server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 testing.go:149: server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: 2021-10-08T13:01:07.549Z [WARN] server-737be045-7aa9-8b68-fc51-d9cea4d8cae1: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [WARN] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: appendEntries rejected, sending older logs: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" next=1 testing.go:149: cluster: 2021-10-08T13:01:07.549Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.549Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: pipelining replication: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" testing.go:149: cluster: 2021-10-08T13:01:07.655Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.549771164 +0000 UTC m=+7.564453899 (1 nodes), 907ns from start of poll, 276.801349ms from cluster start. Timeout at 2021-10-08 13:01:07.654896368 +0000 UTC m=+7.669579113, 105.125214ms after stability testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.655Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: aborting pipeline replication: peer="{Voter server-737be045-7aa9-8b68-fc51-d9cea4d8cae1 737be045-7aa9-8b68-fc51-d9cea4d8cae1}" testing.go:149: server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: 2021-10-08T13:01:07.655Z [INFO] server-2e582ec0-4b59-10e5-4429-3788f5f9ceeb: aborting pipeline replication: peer="{Voter server-e304b84e-17af-a6a8-ff5a-afd243d28d08 e304b84e-17af-a6a8-ff5a-afd243d28d08}" === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.658Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testLoggerAdapter verbose: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.658Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: entering follower state: follower="Node at 6deb3137-6625-8102-d8d1-abb40b801fd4 [Follower]" leader= testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.658Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.658Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering follower state: follower="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Follower]" leader= testLoggerAdapter verbose: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.658Z [INFO] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.658Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.658Z [INFO] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: entering follower state: follower="Node at 9d6fef01-20df-8262-e227-9ce2b322e8b1 [Follower]" leader= testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [WARN] server-9502afdb-5144-e8e0-680d-bef336173d3b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering candidate state: node="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Candidate]" term=2 testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: election won: tally=2 testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering leader state: leader="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Leader]" testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: added peer, starting replication: peer=server-6deb3137-6625-8102-d8d1-abb40b801fd4 testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: added peer, starting replication: peer=server-9d6fef01-20df-8262-e227-9ce2b322e8b1 testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.712Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: pipelining replication: peer="{Voter server-6deb3137-6625-8102-d8d1-abb40b801fd4 6deb3137-6625-8102-d8d1-abb40b801fd4}" testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.712Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: pipelining replication: peer="{Voter server-9d6fef01-20df-8262-e227-9ce2b322e8b1 9d6fef01-20df-8262-e227-9ce2b322e8b1}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.817Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.711985392 +0000 UTC m=+7.726668126 (1 nodes), 53.148177ms from start of poll, 53.914795ms from cluster start. Timeout at 2021-10-08 13:01:07.817057122 +0000 UTC m=+7.831739845, 105.071719ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.822Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:07.927Z [INFO] stable state for Follower reached at 2021-10-08 13:01:07.822447308 +0000 UTC m=+7.837130043 (2 nodes), 1.165µs from start of poll, 164.376712ms from cluster start. Timeout at 2021-10-08 13:01:07.927520486 +0000 UTC m=+7.942203238, 105.073195ms after stability testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: updating configuration: command=RemoveServer server-id=server-6deb3137-6625-8102-d8d1-abb40b801fd4 server-addr= servers="[{Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: removed peer, stopping replication: peer=server-6deb3137-6625-8102-d8d1-abb40b801fd4 last-index=3 testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: aborting pipeline replication: peer="{Voter server-6deb3137-6625-8102-d8d1-abb40b801fd4 6deb3137-6625-8102-d8d1-abb40b801fd4}" testLoggerAdapter verbose: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.990Z [WARN] server-6deb3137-6625-8102-d8d1-abb40b801fd4: heartbeat timeout reached, starting election: last-leader=9502afdb-5144-e8e0-680d-bef336173d3b testLoggerAdapter verbose: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.990Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: entering candidate state: node="Node at 6deb3137-6625-8102-d8d1-abb40b801fd4 [Candidate]" term=3 testLoggerAdapter verbose: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.990Z [WARN] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: rejecting vote request since we have a leader: from=6deb3137-6625-8102-d8d1-abb40b801fd4 leader=9502afdb-5144-e8e0-680d-bef336173d3b testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.990Z [WARN] server-9502afdb-5144-e8e0-680d-bef336173d3b: rejecting vote request since we have a leader: from=6deb3137-6625-8102-d8d1-abb40b801fd4 leader=9502afdb-5144-e8e0-680d-bef336173d3b testLoggerAdapter verbose: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:08.033Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: aborting pipeline replication: peer="{Voter server-9d6fef01-20df-8262-e227-9ce2b322e8b1 9d6fef01-20df-8262-e227-9ce2b322e8b1}" --- PASS: TestRaft_RemoveFollower (0.38s) testing.go:149: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.658Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testing.go:149: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.658Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: entering follower state: follower="Node at 6deb3137-6625-8102-d8d1-abb40b801fd4 [Follower]" leader= testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.658Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.658Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering follower state: follower="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Follower]" leader= testing.go:149: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.658Z [INFO] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6deb3137-6625-8102-d8d1-abb40b801fd4 Address:6deb3137-6625-8102-d8d1-abb40b801fd4} {Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testing.go:149: cluster: 2021-10-08T13:01:07.658Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.658Z [INFO] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: entering follower state: follower="Node at 9d6fef01-20df-8262-e227-9ce2b322e8b1 [Follower]" leader= testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [WARN] server-9502afdb-5144-e8e0-680d-bef336173d3b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering candidate state: node="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Candidate]" term=2 testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: election won: tally=2 testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: entering leader state: leader="Node at 9502afdb-5144-e8e0-680d-bef336173d3b [Leader]" testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: added peer, starting replication: peer=server-6deb3137-6625-8102-d8d1-abb40b801fd4 testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.711Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: added peer, starting replication: peer=server-9d6fef01-20df-8262-e227-9ce2b322e8b1 testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.712Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: pipelining replication: peer="{Voter server-6deb3137-6625-8102-d8d1-abb40b801fd4 6deb3137-6625-8102-d8d1-abb40b801fd4}" testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.712Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: pipelining replication: peer="{Voter server-9d6fef01-20df-8262-e227-9ce2b322e8b1 9d6fef01-20df-8262-e227-9ce2b322e8b1}" testing.go:149: cluster: 2021-10-08T13:01:07.817Z [INFO] stable state for Leader reached at 2021-10-08 13:01:07.711985392 +0000 UTC m=+7.726668126 (1 nodes), 53.148177ms from start of poll, 53.914795ms from cluster start. Timeout at 2021-10-08 13:01:07.817057122 +0000 UTC m=+7.831739845, 105.071719ms after stability testing.go:149: cluster: 2021-10-08T13:01:07.822Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:07.927Z [INFO] stable state for Follower reached at 2021-10-08 13:01:07.822447308 +0000 UTC m=+7.837130043 (2 nodes), 1.165µs from start of poll, 164.376712ms from cluster start. Timeout at 2021-10-08 13:01:07.927520486 +0000 UTC m=+7.942203238, 105.073195ms after stability testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: updating configuration: command=RemoveServer server-id=server-6deb3137-6625-8102-d8d1-abb40b801fd4 server-addr= servers="[{Suffrage:Voter ID:server-9502afdb-5144-e8e0-680d-bef336173d3b Address:9502afdb-5144-e8e0-680d-bef336173d3b} {Suffrage:Voter ID:server-9d6fef01-20df-8262-e227-9ce2b322e8b1 Address:9d6fef01-20df-8262-e227-9ce2b322e8b1}]" testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: removed peer, stopping replication: peer=server-6deb3137-6625-8102-d8d1-abb40b801fd4 last-index=3 testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.927Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: aborting pipeline replication: peer="{Voter server-6deb3137-6625-8102-d8d1-abb40b801fd4 6deb3137-6625-8102-d8d1-abb40b801fd4}" testing.go:149: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.990Z [WARN] server-6deb3137-6625-8102-d8d1-abb40b801fd4: heartbeat timeout reached, starting election: last-leader=9502afdb-5144-e8e0-680d-bef336173d3b testing.go:149: server-6deb3137-6625-8102-d8d1-abb40b801fd4: 2021-10-08T13:01:07.990Z [INFO] server-6deb3137-6625-8102-d8d1-abb40b801fd4: entering candidate state: node="Node at 6deb3137-6625-8102-d8d1-abb40b801fd4 [Candidate]" term=3 testing.go:149: server-9d6fef01-20df-8262-e227-9ce2b322e8b1: 2021-10-08T13:01:07.990Z [WARN] server-9d6fef01-20df-8262-e227-9ce2b322e8b1: rejecting vote request since we have a leader: from=6deb3137-6625-8102-d8d1-abb40b801fd4 leader=9502afdb-5144-e8e0-680d-bef336173d3b testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:07.990Z [WARN] server-9502afdb-5144-e8e0-680d-bef336173d3b: rejecting vote request since we have a leader: from=6deb3137-6625-8102-d8d1-abb40b801fd4 leader=9502afdb-5144-e8e0-680d-bef336173d3b testing.go:149: server-9502afdb-5144-e8e0-680d-bef336173d3b: 2021-10-08T13:01:08.033Z [INFO] server-9502afdb-5144-e8e0-680d-bef336173d3b: aborting pipeline replication: peer="{Voter server-9d6fef01-20df-8262-e227-9ce2b322e8b1 9d6fef01-20df-8262-e227-9ce2b322e8b1}" === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.035Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.035Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering follower state: follower="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Follower]" leader= testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.035Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.035Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering follower state: follower="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Follower]" leader= testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.035Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.035Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.035Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering follower state: follower="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Follower]" leader= testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [WARN] server-a3239638-d428-287e-dc71-cacc58f83c2f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering candidate state: node="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Candidate]" term=2 testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: election won: tally=2 testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering leader state: leader="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Leader]" testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: added peer, starting replication: peer=server-4cd538e1-b644-554b-c421-4b76262d70d8 testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: added peer, starting replication: peer=server-579ad806-efe5-2e73-3594-79bb600c4852 testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: pipelining replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: pipelining replication: peer="{Voter server-4cd538e1-b644-554b-c421-4b76262d70d8 4cd538e1-b644-554b-c421-4b76262d70d8}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.201Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.096648477 +0000 UTC m=+8.111331239 (1 nodes), 60.736982ms from start of poll, 61.501205ms from cluster start. Timeout at 2021-10-08 13:01:08.201714091 +0000 UTC m=+8.216396814, 105.065575ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.207Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.312Z [INFO] stable state for Follower reached at 2021-10-08 13:01:08.207061264 +0000 UTC m=+8.221743992 (2 nodes), 1.443µs from start of poll, 171.913958ms from cluster start. Timeout at 2021-10-08 13:01:08.312115316 +0000 UTC m=+8.326798040, 105.054048ms after stability testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: updating configuration: command=RemoveServer server-id=server-a3239638-d428-287e-dc71-cacc58f83c2f server-addr= servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852}]" testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: removed ourself, shutting down testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: aborting pipeline replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testLoggerAdapter verbose: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: aborting pipeline replication: peer="{Voter server-4cd538e1-b644-554b-c421-4b76262d70d8 4cd538e1-b644-554b-c421-4b76262d70d8}" testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.410Z [WARN] server-4cd538e1-b644-554b-c421-4b76262d70d8: heartbeat timeout reached, starting election: last-leader=a3239638-d428-287e-dc71-cacc58f83c2f testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.410Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering candidate state: node="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Candidate]" term=3 testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.410Z [WARN] server-579ad806-efe5-2e73-3594-79bb600c4852: rejecting vote request since we have a leader: from=4cd538e1-b644-554b-c421-4b76262d70d8 leader=a3239638-d428-287e-dc71-cacc58f83c2f testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.450Z [WARN] server-579ad806-efe5-2e73-3594-79bb600c4852: heartbeat timeout reached, starting election: last-leader=a3239638-d428-287e-dc71-cacc58f83c2f testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.450Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering candidate state: node="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Candidate]" term=3 testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.450Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [WARN] server-4cd538e1-b644-554b-c421-4b76262d70d8: Election timeout reached, restarting election testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering candidate state: node="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Candidate]" term=4 testLoggerAdapter verbose: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.498Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering follower state: follower="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Follower]" leader= testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: election won: tally=2 testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering leader state: leader="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Leader]" testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: added peer, starting replication: peer=server-579ad806-efe5-2e73-3594-79bb600c4852 testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.499Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: pipelining replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.522Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.628Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.522978732 +0000 UTC m=+8.537661453 (1 nodes), 24.817µs from start of poll, 487.831419ms from cluster start. Timeout at 2021-10-08 13:01:08.628097224 +0000 UTC m=+8.642779963, 105.11851ms after stability testLoggerAdapter verbose: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.628Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: aborting pipeline replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" --- PASS: TestRaft_RemoveLeader (0.59s) testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.035Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.035Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering follower state: follower="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Follower]" leader= testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.035Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.035Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering follower state: follower="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Follower]" leader= testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.035Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852} {Suffrage:Voter ID:server-a3239638-d428-287e-dc71-cacc58f83c2f Address:a3239638-d428-287e-dc71-cacc58f83c2f}]" testing.go:149: cluster: 2021-10-08T13:01:08.035Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.035Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering follower state: follower="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Follower]" leader= testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [WARN] server-a3239638-d428-287e-dc71-cacc58f83c2f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering candidate state: node="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Candidate]" term=2 testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: election won: tally=2 testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: entering leader state: leader="Node at a3239638-d428-287e-dc71-cacc58f83c2f [Leader]" testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: added peer, starting replication: peer=server-4cd538e1-b644-554b-c421-4b76262d70d8 testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: added peer, starting replication: peer=server-579ad806-efe5-2e73-3594-79bb600c4852 testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: pipelining replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.096Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: pipelining replication: peer="{Voter server-4cd538e1-b644-554b-c421-4b76262d70d8 4cd538e1-b644-554b-c421-4b76262d70d8}" testing.go:149: cluster: 2021-10-08T13:01:08.201Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.096648477 +0000 UTC m=+8.111331239 (1 nodes), 60.736982ms from start of poll, 61.501205ms from cluster start. Timeout at 2021-10-08 13:01:08.201714091 +0000 UTC m=+8.216396814, 105.065575ms after stability testing.go:149: cluster: 2021-10-08T13:01:08.207Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:08.312Z [INFO] stable state for Follower reached at 2021-10-08 13:01:08.207061264 +0000 UTC m=+8.221743992 (2 nodes), 1.443µs from start of poll, 171.913958ms from cluster start. Timeout at 2021-10-08 13:01:08.312115316 +0000 UTC m=+8.326798040, 105.054048ms after stability testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: updating configuration: command=RemoveServer server-id=server-a3239638-d428-287e-dc71-cacc58f83c2f server-addr= servers="[{Suffrage:Voter ID:server-4cd538e1-b644-554b-c421-4b76262d70d8 Address:4cd538e1-b644-554b-c421-4b76262d70d8} {Suffrage:Voter ID:server-579ad806-efe5-2e73-3594-79bb600c4852 Address:579ad806-efe5-2e73-3594-79bb600c4852}]" testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: removed ourself, shutting down testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: aborting pipeline replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testing.go:149: server-a3239638-d428-287e-dc71-cacc58f83c2f: 2021-10-08T13:01:08.312Z [INFO] server-a3239638-d428-287e-dc71-cacc58f83c2f: aborting pipeline replication: peer="{Voter server-4cd538e1-b644-554b-c421-4b76262d70d8 4cd538e1-b644-554b-c421-4b76262d70d8}" testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.410Z [WARN] server-4cd538e1-b644-554b-c421-4b76262d70d8: heartbeat timeout reached, starting election: last-leader=a3239638-d428-287e-dc71-cacc58f83c2f testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.410Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering candidate state: node="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Candidate]" term=3 testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.410Z [WARN] server-579ad806-efe5-2e73-3594-79bb600c4852: rejecting vote request since we have a leader: from=4cd538e1-b644-554b-c421-4b76262d70d8 leader=a3239638-d428-287e-dc71-cacc58f83c2f testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.450Z [WARN] server-579ad806-efe5-2e73-3594-79bb600c4852: heartbeat timeout reached, starting election: last-leader=a3239638-d428-287e-dc71-cacc58f83c2f testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.450Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering candidate state: node="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Candidate]" term=3 testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.450Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: duplicate requestVote for same term: term=3 testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [WARN] server-4cd538e1-b644-554b-c421-4b76262d70d8: Election timeout reached, restarting election testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering candidate state: node="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Candidate]" term=4 testing.go:149: server-579ad806-efe5-2e73-3594-79bb600c4852: 2021-10-08T13:01:08.498Z [INFO] server-579ad806-efe5-2e73-3594-79bb600c4852: entering follower state: follower="Node at 579ad806-efe5-2e73-3594-79bb600c4852 [Follower]" leader= testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: election won: tally=2 testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: entering leader state: leader="Node at 4cd538e1-b644-554b-c421-4b76262d70d8 [Leader]" testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.498Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: added peer, starting replication: peer=server-579ad806-efe5-2e73-3594-79bb600c4852 testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.499Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: pipelining replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" testing.go:149: cluster: 2021-10-08T13:01:08.522Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:08.628Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.522978732 +0000 UTC m=+8.537661453 (1 nodes), 24.817µs from start of poll, 487.831419ms from cluster start. Timeout at 2021-10-08 13:01:08.628097224 +0000 UTC m=+8.642779963, 105.11851ms after stability testing.go:149: server-4cd538e1-b644-554b-c421-4b76262d70d8: 2021-10-08T13:01:08.628Z [INFO] server-4cd538e1-b644-554b-c421-4b76262d70d8: aborting pipeline replication: peer="{Voter server-579ad806-efe5-2e73-3594-79bb600c4852 579ad806-efe5-2e73-3594-79bb600c4852}" === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.629Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.629Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.629Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering follower state: follower="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Follower]" leader= testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.630Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering follower state: follower="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Follower]" leader= testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.630Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.630Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.630Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering follower state: follower="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Follower]" leader= testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.680Z [WARN] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.680Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering candidate state: node="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Candidate]" term=2 testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: election won: tally=2 testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering leader state: leader="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Leader]" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: added peer, starting replication: peer=server-b046f1b6-5dfe-1750-c060-b1211f78aa1a testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: added peer, starting replication: peer=server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: pipelining replication: peer="{Voter server-b046f1b6-5dfe-1750-c060-b1211f78aa1a b046f1b6-5dfe-1750-c060-b1211f78aa1a}" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: pipelining replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.786Z [INFO] stable state for Follower reached at 2021-10-08 13:01:08.681160885 +0000 UTC m=+8.695843618 (2 nodes), 50.991737ms from start of poll, 51.429372ms from cluster start. Timeout at 2021-10-08 13:01:08.786246282 +0000 UTC m=+8.800929025, 105.085407ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.786Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.891Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.786404814 +0000 UTC m=+8.801087541 (1 nodes), 944ns from start of poll, 156.673295ms from cluster start. Timeout at 2021-10-08 13:01:08.891523244 +0000 UTC m=+8.906205991, 105.11845ms after stability testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.893Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: updating configuration: command=RemoveServer server-id=server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 server-addr= servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.893Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: removed ourself, transitioning to follower testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering follower state: follower="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Follower]" leader= testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: aborting pipeline replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: aborting pipeline replication: peer="{Voter server-b046f1b6-5dfe-1750-c060-b1211f78aa1a b046f1b6-5dfe-1750-c060-b1211f78aa1a}" testLoggerAdapter verbose: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.963Z [WARN] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: not part of stable configuration, aborting election testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.990Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: heartbeat timeout reached, starting election: last-leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.990Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=3 testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.990Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: rejecting vote request since we have a leader: from=b046f1b6-5dfe-1750-c060-b1211f78aa1a leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:08.999Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.006Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: heartbeat timeout reached, starting election: last-leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.006Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering candidate state: node="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Candidate]" term=3 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.006Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: Election timeout reached, restarting election testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=4 testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: Election timeout reached, restarting election testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering candidate state: node="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Candidate]" term=4 testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: duplicate requestVote for same term: term=4 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: duplicate requestVote for same term: term=4 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: Election timeout reached, restarting election testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=5 testLoggerAdapter verbose: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.122Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering follower state: follower="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Follower]" leader= testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: election won: tally=2 testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering leader state: leader="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Leader]" testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: added peer, starting replication: peer=server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: pipelining replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.227Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.122735546 +0000 UTC m=+9.137418307 (1 nodes), 123.277333ms from start of poll, 493.004061ms from cluster start. Timeout at 2021-10-08 13:01:09.227818622 +0000 UTC m=+9.242501377, 105.08307ms after stability testLoggerAdapter verbose: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.333Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: aborting pipeline replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.71s) testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.629Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.629Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.629Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering follower state: follower="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Follower]" leader= testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.630Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering follower state: follower="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Follower]" leader= testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.630Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 Address:6448cd0e-81e6-0c2a-029c-3012b9cd7c82} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testing.go:149: cluster: 2021-10-08T13:01:08.630Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.630Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering follower state: follower="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Follower]" leader= testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.680Z [WARN] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.680Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering candidate state: node="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Candidate]" term=2 testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: election won: tally=2 testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering leader state: leader="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Leader]" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: added peer, starting replication: peer=server-b046f1b6-5dfe-1750-c060-b1211f78aa1a testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: added peer, starting replication: peer=server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: pipelining replication: peer="{Voter server-b046f1b6-5dfe-1750-c060-b1211f78aa1a b046f1b6-5dfe-1750-c060-b1211f78aa1a}" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.681Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: pipelining replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testing.go:149: cluster: 2021-10-08T13:01:08.786Z [INFO] stable state for Follower reached at 2021-10-08 13:01:08.681160885 +0000 UTC m=+8.695843618 (2 nodes), 50.991737ms from start of poll, 51.429372ms from cluster start. Timeout at 2021-10-08 13:01:08.786246282 +0000 UTC m=+8.800929025, 105.085407ms after stability testing.go:149: cluster: 2021-10-08T13:01:08.786Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:08.891Z [INFO] stable state for Leader reached at 2021-10-08 13:01:08.786404814 +0000 UTC m=+8.801087541 (1 nodes), 944ns from start of poll, 156.673295ms from cluster start. Timeout at 2021-10-08 13:01:08.891523244 +0000 UTC m=+8.906205991, 105.11845ms after stability testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.893Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: updating configuration: command=RemoveServer server-id=server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82 server-addr= servers="[{Suffrage:Voter ID:server-b046f1b6-5dfe-1750-c060-b1211f78aa1a Address:b046f1b6-5dfe-1750-c060-b1211f78aa1a} {Suffrage:Voter ID:server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f Address:51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}]" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.893Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: removed ourself, transitioning to follower testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: entering follower state: follower="Node at 6448cd0e-81e6-0c2a-029c-3012b9cd7c82 [Follower]" leader= testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: aborting pipeline replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.894Z [INFO] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: aborting pipeline replication: peer="{Voter server-b046f1b6-5dfe-1750-c060-b1211f78aa1a b046f1b6-5dfe-1750-c060-b1211f78aa1a}" testing.go:149: server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: 2021-10-08T13:01:08.963Z [WARN] server-6448cd0e-81e6-0c2a-029c-3012b9cd7c82: not part of stable configuration, aborting election testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.990Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: heartbeat timeout reached, starting election: last-leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:08.990Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=3 testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:08.990Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: rejecting vote request since we have a leader: from=b046f1b6-5dfe-1750-c060-b1211f78aa1a leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testing.go:149: cluster: 2021-10-08T13:01:08.999Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.006Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: heartbeat timeout reached, starting election: last-leader=6448cd0e-81e6-0c2a-029c-3012b9cd7c82 testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.006Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering candidate state: node="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Candidate]" term=3 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.006Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: duplicate requestVote for same term: term=3 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: Election timeout reached, restarting election testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=4 testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [WARN] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: Election timeout reached, restarting election testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering candidate state: node="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Candidate]" term=4 testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.062Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: duplicate requestVote for same term: term=4 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.062Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: duplicate requestVote for same term: term=4 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [WARN] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: Election timeout reached, restarting election testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering candidate state: node="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Candidate]" term=5 testing.go:149: server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: 2021-10-08T13:01:09.122Z [INFO] server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f: entering follower state: follower="Node at 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f [Follower]" leader= testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: election won: tally=2 testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: entering leader state: leader="Node at b046f1b6-5dfe-1750-c060-b1211f78aa1a [Leader]" testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: added peer, starting replication: peer=server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.122Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: pipelining replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" testing.go:149: cluster: 2021-10-08T13:01:09.227Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.122735546 +0000 UTC m=+9.137418307 (1 nodes), 123.277333ms from start of poll, 493.004061ms from cluster start. Timeout at 2021-10-08 13:01:09.227818622 +0000 UTC m=+9.242501377, 105.08307ms after stability testing.go:149: server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: 2021-10-08T13:01:09.333Z [INFO] server-b046f1b6-5dfe-1750-c060-b1211f78aa1a: aborting pipeline replication: peer="{Voter server-51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f 51751fc7-c3ff-47f0-55ee-0af6cc8b3a0f}" === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: 2021-10-08T13:01:09.335Z [INFO] server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testLoggerAdapter verbose: server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: 2021-10-08T13:01:09.335Z [INFO] server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: entering follower state: follower="Node at 2d723dd4-a075-f329-7d38-9d86d5a76c1f [Follower]" leader= testLoggerAdapter verbose: server-9934bfa1-c927-dbd1-97e0-284895106c99: 2021-10-08T13:01:09.335Z [INFO] server-9934bfa1-c927-dbd1-97e0-284895106c99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testLoggerAdapter verbose: server-9934bfa1-c927-dbd1-97e0-284895106c99: 2021-10-08T13:01:09.335Z [INFO] server-9934bfa1-c927-dbd1-97e0-284895106c99: entering follower state: follower="Node at 9934bfa1-c927-dbd1-97e0-284895106c99 [Follower]" leader= testLoggerAdapter verbose: server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: 2021-10-08T13:01:09.336Z [INFO] server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testLoggerAdapter verbose: server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: 2021-10-08T13:01:09.336Z [INFO] server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: entering follower state: follower="Node at 3ce5d159-1bf2-af5a-353c-fb4cd8418326 [Follower]" leader= testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.336Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.336Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.336Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering follower state: follower="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Follower]" leader= testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering candidate state: node="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Candidate]" term=2 testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: election won: tally=3 testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering leader state: leader="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Leader]" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-2d723dd4-a075-f329-7d38-9d86d5a76c1f testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-9934bfa1-c927-dbd1-97e0-284895106c99 testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 3ce5d159-1bf2-af5a-353c-fb4cd8418326}" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.501Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.395824515 +0000 UTC m=+9.410507247 (1 nodes), 59.458807ms from start of poll, 60.386462ms from cluster start. Timeout at 2021-10-08 13:01:09.500922634 +0000 UTC m=+9.515605358, 105.098111ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.606Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.711Z [INFO] stable state for Follower reached at 2021-10-08 13:01:09.606287922 +0000 UTC m=+9.620970643 (3 nodes), 840ns from start of poll, 270.849858ms from cluster start. Timeout at 2021-10-08 13:01:09.711362888 +0000 UTC m=+9.726045622, 105.074979ms after stability testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: updating configuration: command=RemoveServer server-id=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 server-addr= servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: removed peer, stopping replication: peer=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 last-index=3 testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to pipeline appendEntries: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="append pipeline closed" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to pipeline appendEntries: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="append pipeline closed" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 3ce5d159-1bf2-af5a-353c-fb4cd8418326}" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.714Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.717Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.720Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.721Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.732Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.734Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.738Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.740Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.750Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.752Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.756Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.756Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-9934bfa1-c927-dbd1-97e0-284895106c99 time=50.139241ms testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.757Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-2d723dd4-a075-f329-7d38-9d86d5a76c1f time=55.65693ms testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-9934bfa1-c927-dbd1-97e0-284895106c99 time=60.470169ms testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering follower state: follower="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Follower]" leader= testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testLoggerAdapter verbose: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.43s) testing.go:149: server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: 2021-10-08T13:01:09.335Z [INFO] server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testing.go:149: server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: 2021-10-08T13:01:09.335Z [INFO] server-2d723dd4-a075-f329-7d38-9d86d5a76c1f: entering follower state: follower="Node at 2d723dd4-a075-f329-7d38-9d86d5a76c1f [Follower]" leader= testing.go:149: server-9934bfa1-c927-dbd1-97e0-284895106c99: 2021-10-08T13:01:09.335Z [INFO] server-9934bfa1-c927-dbd1-97e0-284895106c99: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testing.go:149: server-9934bfa1-c927-dbd1-97e0-284895106c99: 2021-10-08T13:01:09.335Z [INFO] server-9934bfa1-c927-dbd1-97e0-284895106c99: entering follower state: follower="Node at 9934bfa1-c927-dbd1-97e0-284895106c99 [Follower]" leader= testing.go:149: server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: 2021-10-08T13:01:09.336Z [INFO] server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testing.go:149: server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: 2021-10-08T13:01:09.336Z [INFO] server-3ce5d159-1bf2-af5a-353c-fb4cd8418326: entering follower state: follower="Node at 3ce5d159-1bf2-af5a-353c-fb4cd8418326 [Follower]" leader= testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.336Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 Address:3ce5d159-1bf2-af5a-353c-fb4cd8418326} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testing.go:149: cluster: 2021-10-08T13:01:09.336Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.336Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering follower state: follower="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Follower]" leader= testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering candidate state: node="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Candidate]" term=2 testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: election won: tally=3 testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering leader state: leader="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Leader]" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-2d723dd4-a075-f329-7d38-9d86d5a76c1f testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-9934bfa1-c927-dbd1-97e0-284895106c99 testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: added peer, starting replication: peer=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 3ce5d159-1bf2-af5a-353c-fb4cd8418326}" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.395Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: pipelining replication: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" testing.go:149: cluster: 2021-10-08T13:01:09.501Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.395824515 +0000 UTC m=+9.410507247 (1 nodes), 59.458807ms from start of poll, 60.386462ms from cluster start. Timeout at 2021-10-08 13:01:09.500922634 +0000 UTC m=+9.515605358, 105.098111ms after stability testing.go:149: cluster: 2021-10-08T13:01:09.606Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:09.711Z [INFO] stable state for Follower reached at 2021-10-08 13:01:09.606287922 +0000 UTC m=+9.620970643 (3 nodes), 840ns from start of poll, 270.849858ms from cluster start. Timeout at 2021-10-08 13:01:09.711362888 +0000 UTC m=+9.726045622, 105.074979ms after stability testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: updating configuration: command=RemoveServer server-id=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 server-addr= servers="[{Suffrage:Voter ID:server-2d723dd4-a075-f329-7d38-9d86d5a76c1f Address:2d723dd4-a075-f329-7d38-9d86d5a76c1f} {Suffrage:Voter ID:server-9934bfa1-c927-dbd1-97e0-284895106c99 Address:9934bfa1-c927-dbd1-97e0-284895106c99} {Suffrage:Voter ID:server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 Address:05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450}]" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: removed peer, stopping replication: peer=server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 last-index=3 testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to pipeline appendEntries: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="append pipeline closed" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to pipeline appendEntries: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="append pipeline closed" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.711Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: aborting pipeline replication: peer="{Voter server-3ce5d159-1bf2-af5a-353c-fb4cd8418326 3ce5d159-1bf2-af5a-353c-fb4cd8418326}" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.714Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.717Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.720Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.721Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.732Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.734Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.738Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.740Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.750Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=2d723dd4-a075-f329-7d38-9d86d5a76c1f error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.752Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to heartbeat to: peer=9934bfa1-c927-dbd1-97e0-284895106c99 error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.756Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.756Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-9934bfa1-c927-dbd1-97e0-284895106c99 time=50.139241ms testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.757Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-2d723dd4-a075-f329-7d38-9d86d5a76c1f time=55.65693ms testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact: server-id=server-9934bfa1-c927-dbd1-97e0-284895106c99 time=60.470169ms testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [WARN] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to contact quorum of nodes, stepping down testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [INFO] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: entering follower state: follower="Node at 05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450 [Follower]" leader= testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-2d723dd4-a075-f329-7d38-9d86d5a76c1f 2d723dd4-a075-f329-7d38-9d86d5a76c1f}" error="failed to connect to peer: 2d723dd4-a075-f329-7d38-9d86d5a76c1f" testing.go:149: server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: 2021-10-08T13:01:09.766Z [ERROR] server-05aaceb2-f6ee-c0f0-6e8e-4ba79ddae450: failed to appendEntries to: peer="{Voter server-9934bfa1-c927-dbd1-97e0-284895106c99 9934bfa1-c927-dbd1-97e0-284895106c99}" error="failed to connect to peer: 9934bfa1-c927-dbd1-97e0-284895106c99" === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-da90f3d3-97d1-0cc8-beab-8face63d9902: 2021-10-08T13:01:09.768Z [INFO] server-da90f3d3-97d1-0cc8-beab-8face63d9902: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.768Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testLoggerAdapter verbose: server-da90f3d3-97d1-0cc8-beab-8face63d9902: 2021-10-08T13:01:09.768Z [INFO] server-da90f3d3-97d1-0cc8-beab-8face63d9902: entering follower state: follower="Node at da90f3d3-97d1-0cc8-beab-8face63d9902 [Follower]" leader= testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.768Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering follower state: follower="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Follower]" leader= testLoggerAdapter verbose: server-60690406-703d-8a9f-0bc3-68d35b6b5104: 2021-10-08T13:01:09.769Z [INFO] server-60690406-703d-8a9f-0bc3-68d35b6b5104: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.769Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-60690406-703d-8a9f-0bc3-68d35b6b5104: 2021-10-08T13:01:09.769Z [INFO] server-60690406-703d-8a9f-0bc3-68d35b6b5104: entering follower state: follower="Node at 60690406-703d-8a9f-0bc3-68d35b6b5104 [Follower]" leader= testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [WARN] server-01c848a7-72cb-1bef-d9de-1962ee95caac: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering candidate state: node="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Candidate]" term=2 testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: election won: tally=2 testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering leader state: leader="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Leader]" testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: added peer, starting replication: peer=server-da90f3d3-97d1-0cc8-beab-8face63d9902 testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: added peer, starting replication: peer=server-60690406-703d-8a9f-0bc3-68d35b6b5104 testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: pipelining replication: peer="{Voter server-da90f3d3-97d1-0cc8-beab-8face63d9902 da90f3d3-97d1-0cc8-beab-8face63d9902}" testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: pipelining replication: peer="{Voter server-60690406-703d-8a9f-0bc3-68d35b6b5104 60690406-703d-8a9f-0bc3-68d35b6b5104}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.939Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.834471713 +0000 UTC m=+9.849154482 (1 nodes), 64.823021ms from start of poll, 66.199311ms from cluster start. Timeout at 2021-10-08 13:01:09.939569385 +0000 UTC m=+9.954252127, 105.097645ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:09.939Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.044Z [INFO] stable state for Follower reached at 2021-10-08 13:01:09.939721433 +0000 UTC m=+9.954404167 (2 nodes), 1.037µs from start of poll, 171.448996ms from cluster start. Timeout at 2021-10-08 13:01:10.044815203 +0000 UTC m=+10.059497948, 105.093781ms after stability testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: updating configuration: command=AddStaging server-id=server-da90f3d3-97d1-0cc8-beab-8face63d9902 server-addr=da90f3d3-97d1-0cc8-beab-8face63d9902 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: aborting pipeline replication: peer="{Voter server-60690406-703d-8a9f-0bc3-68d35b6b5104 60690406-703d-8a9f-0bc3-68d35b6b5104}" testLoggerAdapter verbose: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: aborting pipeline replication: peer="{Voter server-da90f3d3-97d1-0cc8-beab-8face63d9902 da90f3d3-97d1-0cc8-beab-8face63d9902}" --- PASS: TestRaft_AddKnownPeer (0.28s) testing.go:149: server-da90f3d3-97d1-0cc8-beab-8face63d9902: 2021-10-08T13:01:09.768Z [INFO] server-da90f3d3-97d1-0cc8-beab-8face63d9902: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.768Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testing.go:149: server-da90f3d3-97d1-0cc8-beab-8face63d9902: 2021-10-08T13:01:09.768Z [INFO] server-da90f3d3-97d1-0cc8-beab-8face63d9902: entering follower state: follower="Node at da90f3d3-97d1-0cc8-beab-8face63d9902 [Follower]" leader= testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.768Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering follower state: follower="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Follower]" leader= testing.go:149: server-60690406-703d-8a9f-0bc3-68d35b6b5104: 2021-10-08T13:01:09.769Z [INFO] server-60690406-703d-8a9f-0bc3-68d35b6b5104: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testing.go:149: cluster: 2021-10-08T13:01:09.769Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-60690406-703d-8a9f-0bc3-68d35b6b5104: 2021-10-08T13:01:09.769Z [INFO] server-60690406-703d-8a9f-0bc3-68d35b6b5104: entering follower state: follower="Node at 60690406-703d-8a9f-0bc3-68d35b6b5104 [Follower]" leader= testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [WARN] server-01c848a7-72cb-1bef-d9de-1962ee95caac: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering candidate state: node="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Candidate]" term=2 testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: election won: tally=2 testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: entering leader state: leader="Node at 01c848a7-72cb-1bef-d9de-1962ee95caac [Leader]" testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: added peer, starting replication: peer=server-da90f3d3-97d1-0cc8-beab-8face63d9902 testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: added peer, starting replication: peer=server-60690406-703d-8a9f-0bc3-68d35b6b5104 testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: pipelining replication: peer="{Voter server-da90f3d3-97d1-0cc8-beab-8face63d9902 da90f3d3-97d1-0cc8-beab-8face63d9902}" testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:09.834Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: pipelining replication: peer="{Voter server-60690406-703d-8a9f-0bc3-68d35b6b5104 60690406-703d-8a9f-0bc3-68d35b6b5104}" testing.go:149: cluster: 2021-10-08T13:01:09.939Z [INFO] stable state for Leader reached at 2021-10-08 13:01:09.834471713 +0000 UTC m=+9.849154482 (1 nodes), 64.823021ms from start of poll, 66.199311ms from cluster start. Timeout at 2021-10-08 13:01:09.939569385 +0000 UTC m=+9.954252127, 105.097645ms after stability testing.go:149: cluster: 2021-10-08T13:01:09.939Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:10.044Z [INFO] stable state for Follower reached at 2021-10-08 13:01:09.939721433 +0000 UTC m=+9.954404167 (2 nodes), 1.037µs from start of poll, 171.448996ms from cluster start. Timeout at 2021-10-08 13:01:10.044815203 +0000 UTC m=+10.059497948, 105.093781ms after stability testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: updating configuration: command=AddStaging server-id=server-da90f3d3-97d1-0cc8-beab-8face63d9902 server-addr=da90f3d3-97d1-0cc8-beab-8face63d9902 servers="[{Suffrage:Voter ID:server-da90f3d3-97d1-0cc8-beab-8face63d9902 Address:da90f3d3-97d1-0cc8-beab-8face63d9902} {Suffrage:Voter ID:server-01c848a7-72cb-1bef-d9de-1962ee95caac Address:01c848a7-72cb-1bef-d9de-1962ee95caac} {Suffrage:Voter ID:server-60690406-703d-8a9f-0bc3-68d35b6b5104 Address:60690406-703d-8a9f-0bc3-68d35b6b5104}]" testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: aborting pipeline replication: peer="{Voter server-60690406-703d-8a9f-0bc3-68d35b6b5104 60690406-703d-8a9f-0bc3-68d35b6b5104}" testing.go:149: server-01c848a7-72cb-1bef-d9de-1962ee95caac: 2021-10-08T13:01:10.045Z [INFO] server-01c848a7-72cb-1bef-d9de-1962ee95caac: aborting pipeline replication: peer="{Voter server-da90f3d3-97d1-0cc8-beab-8face63d9902 da90f3d3-97d1-0cc8-beab-8face63d9902}" === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-50b65039-4a2e-d685-c5da-80c36bc1b037: 2021-10-08T13:01:10.047Z [INFO] server-50b65039-4a2e-d685-c5da-80c36bc1b037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testLoggerAdapter verbose: server-50b65039-4a2e-d685-c5da-80c36bc1b037: 2021-10-08T13:01:10.047Z [INFO] server-50b65039-4a2e-d685-c5da-80c36bc1b037: entering follower state: follower="Node at 50b65039-4a2e-d685-c5da-80c36bc1b037 [Follower]" leader= testLoggerAdapter verbose: server-611020b6-5d80-826b-705f-87589c4f379e: 2021-10-08T13:01:10.047Z [INFO] server-611020b6-5d80-826b-705f-87589c4f379e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testLoggerAdapter verbose: server-611020b6-5d80-826b-705f-87589c4f379e: 2021-10-08T13:01:10.047Z [INFO] server-611020b6-5d80-826b-705f-87589c4f379e: entering follower state: follower="Node at 611020b6-5d80-826b-705f-87589c4f379e [Follower]" leader= testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.047Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.047Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.047Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering follower state: follower="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Follower]" leader= testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [WARN] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering candidate state: node="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Candidate]" term=2 testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: election won: tally=2 testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering leader state: leader="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Leader]" testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: added peer, starting replication: peer=server-50b65039-4a2e-d685-c5da-80c36bc1b037 testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: added peer, starting replication: peer=server-611020b6-5d80-826b-705f-87589c4f379e testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: pipelining replication: peer="{Voter server-611020b6-5d80-826b-705f-87589c4f379e 611020b6-5d80-826b-705f-87589c4f379e}" testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: pipelining replication: peer="{Voter server-50b65039-4a2e-d685-c5da-80c36bc1b037 50b65039-4a2e-d685-c5da-80c36bc1b037}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.205Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.10060801 +0000 UTC m=+10.115290743 (1 nodes), 52.795044ms from start of poll, 53.425317ms from cluster start. Timeout at 2021-10-08 13:01:10.205709228 +0000 UTC m=+10.220391956, 105.101213ms after stability testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.205Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: updating configuration: command=RemoveServer server-id=73d53c00-b7e0-edeb-7e8a-a70f9849280b server-addr= servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.206Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: aborting pipeline replication: peer="{Voter server-611020b6-5d80-826b-705f-87589c4f379e 611020b6-5d80-826b-705f-87589c4f379e}" testLoggerAdapter verbose: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.206Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: aborting pipeline replication: peer="{Voter server-50b65039-4a2e-d685-c5da-80c36bc1b037 50b65039-4a2e-d685-c5da-80c36bc1b037}" --- PASS: TestRaft_RemoveUnknownPeer (0.16s) testing.go:149: server-50b65039-4a2e-d685-c5da-80c36bc1b037: 2021-10-08T13:01:10.047Z [INFO] server-50b65039-4a2e-d685-c5da-80c36bc1b037: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testing.go:149: server-50b65039-4a2e-d685-c5da-80c36bc1b037: 2021-10-08T13:01:10.047Z [INFO] server-50b65039-4a2e-d685-c5da-80c36bc1b037: entering follower state: follower="Node at 50b65039-4a2e-d685-c5da-80c36bc1b037 [Follower]" leader= testing.go:149: server-611020b6-5d80-826b-705f-87589c4f379e: 2021-10-08T13:01:10.047Z [INFO] server-611020b6-5d80-826b-705f-87589c4f379e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testing.go:149: server-611020b6-5d80-826b-705f-87589c4f379e: 2021-10-08T13:01:10.047Z [INFO] server-611020b6-5d80-826b-705f-87589c4f379e: entering follower state: follower="Node at 611020b6-5d80-826b-705f-87589c4f379e [Follower]" leader= testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.047Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testing.go:149: cluster: 2021-10-08T13:01:10.047Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.047Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering follower state: follower="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Follower]" leader= testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [WARN] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering candidate state: node="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Candidate]" term=2 testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: election won: tally=2 testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: entering leader state: leader="Node at f45f8516-3bcc-9930-7bd8-e3cf086da9f2 [Leader]" testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: added peer, starting replication: peer=server-50b65039-4a2e-d685-c5da-80c36bc1b037 testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: added peer, starting replication: peer=server-611020b6-5d80-826b-705f-87589c4f379e testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: pipelining replication: peer="{Voter server-611020b6-5d80-826b-705f-87589c4f379e 611020b6-5d80-826b-705f-87589c4f379e}" testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.100Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: pipelining replication: peer="{Voter server-50b65039-4a2e-d685-c5da-80c36bc1b037 50b65039-4a2e-d685-c5da-80c36bc1b037}" testing.go:149: cluster: 2021-10-08T13:01:10.205Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.10060801 +0000 UTC m=+10.115290743 (1 nodes), 52.795044ms from start of poll, 53.425317ms from cluster start. Timeout at 2021-10-08 13:01:10.205709228 +0000 UTC m=+10.220391956, 105.101213ms after stability testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.205Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: updating configuration: command=RemoveServer server-id=73d53c00-b7e0-edeb-7e8a-a70f9849280b server-addr= servers="[{Suffrage:Voter ID:server-50b65039-4a2e-d685-c5da-80c36bc1b037 Address:50b65039-4a2e-d685-c5da-80c36bc1b037} {Suffrage:Voter ID:server-611020b6-5d80-826b-705f-87589c4f379e Address:611020b6-5d80-826b-705f-87589c4f379e} {Suffrage:Voter ID:server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2 Address:f45f8516-3bcc-9930-7bd8-e3cf086da9f2}]" testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.206Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: aborting pipeline replication: peer="{Voter server-611020b6-5d80-826b-705f-87589c4f379e 611020b6-5d80-826b-705f-87589c4f379e}" testing.go:149: server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: 2021-10-08T13:01:10.206Z [INFO] server-f45f8516-3bcc-9930-7bd8-e3cf086da9f2: aborting pipeline replication: peer="{Voter server-50b65039-4a2e-d685-c5da-80c36bc1b037 50b65039-4a2e-d685-c5da-80c36bc1b037}" === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.207Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f32823-cf2c-5346-bea6-9e29e952e815 Address:33f32823-cf2c-5346-bea6-9e29e952e815}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.207Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.207Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering follower state: follower="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Follower]" leader= testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.272Z [WARN] server-33f32823-cf2c-5346-bea6-9e29e952e815: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering candidate state: node="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Candidate]" term=2 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: election won: tally=1 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering leader state: leader="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.378Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.27316318 +0000 UTC m=+10.287845911 (1 nodes), 65.557945ms from start of poll, 65.828962ms from cluster start. Timeout at 2021-10-08 13:01:10.378237214 +0000 UTC m=+10.392919939, 105.074028ms after stability testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.379Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: starting snapshot up to: index=102 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: snapshot complete up to: index=102 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: restored from snapshot: id=2-102-1633698070379 testLoggerAdapter verbose: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f32823-cf2c-5346-bea6-9e29e952e815 Address:33f32823-cf2c-5346-bea6-9e29e952e815}]" --- PASS: TestRaft_SnapshotRestore (0.18s) testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.207Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f32823-cf2c-5346-bea6-9e29e952e815 Address:33f32823-cf2c-5346-bea6-9e29e952e815}]" testing.go:149: cluster: 2021-10-08T13:01:10.207Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.207Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering follower state: follower="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Follower]" leader= testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.272Z [WARN] server-33f32823-cf2c-5346-bea6-9e29e952e815: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering candidate state: node="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Candidate]" term=2 testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: election won: tally=1 testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.273Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: entering leader state: leader="Node at 33f32823-cf2c-5346-bea6-9e29e952e815 [Leader]" testing.go:149: cluster: 2021-10-08T13:01:10.378Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.27316318 +0000 UTC m=+10.287845911 (1 nodes), 65.557945ms from start of poll, 65.828962ms from cluster start. Timeout at 2021-10-08 13:01:10.378237214 +0000 UTC m=+10.392919939, 105.074028ms after stability testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.379Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: starting snapshot up to: index=102 testing.go:153: 2021-10-08T13:01:10.379Z [INFO] creating new snapshot: path=/tmp/raft836999121/snapshots/2-102-1633698070379.tmp testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: compacting logs: from=1 to=92 testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: snapshot complete up to: index=102 testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: restored from snapshot: id=2-102-1633698070379 testing.go:149: server-33f32823-cf2c-5346-bea6-9e29e952e815: 2021-10-08T13:01:10.382Z [INFO] server-33f32823-cf2c-5346-bea6-9e29e952e815: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-33f32823-cf2c-5346-bea6-9e29e952e815 Address:33f32823-cf2c-5346-bea6-9e29e952e815}]" === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.383Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa248718-eaa9-82ca-9ddf-56833979873e Address:fa248718-eaa9-82ca-9ddf-56833979873e}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.383Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.383Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering follower state: follower="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Follower]" leader= testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [WARN] server-fa248718-eaa9-82ca-9ddf-56833979873e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering candidate state: node="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Candidate]" term=2 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: election won: tally=1 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering leader state: leader="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.562Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.457054869 +0000 UTC m=+10.471737606 (1 nodes), 73.056864ms from start of poll, 73.312866ms from cluster start. Timeout at 2021-10-08 13:01:10.562188656 +0000 UTC m=+10.576871401, 105.133795ms after stability testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.563Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: starting snapshot up to: index=102 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.566Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.566Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: snapshot complete up to: index=102 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.567Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa248718-eaa9-82ca-9ddf-56833979873e Address:fa248718-eaa9-82ca-9ddf-56833979873e}]" --- PASS: TestRaft_NoRestoreOnStart (0.18s) testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.383Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa248718-eaa9-82ca-9ddf-56833979873e Address:fa248718-eaa9-82ca-9ddf-56833979873e}]" testing.go:149: cluster: 2021-10-08T13:01:10.383Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.383Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering follower state: follower="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Follower]" leader= testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [WARN] server-fa248718-eaa9-82ca-9ddf-56833979873e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering candidate state: node="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Candidate]" term=2 testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: election won: tally=1 testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.456Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering leader state: leader="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Leader]" testing.go:149: cluster: 2021-10-08T13:01:10.562Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.457054869 +0000 UTC m=+10.471737606 (1 nodes), 73.056864ms from start of poll, 73.312866ms from cluster start. Timeout at 2021-10-08 13:01:10.562188656 +0000 UTC m=+10.576871401, 105.133795ms after stability testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.563Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: starting snapshot up to: index=102 testing.go:153: 2021-10-08T13:01:10.563Z [INFO] creating new snapshot: path=/tmp/raft258692651/snapshots/2-102-1633698070563.tmp testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.566Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: compacting logs: from=1 to=92 testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.566Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: snapshot complete up to: index=102 testing.go:149: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.567Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fa248718-eaa9-82ca-9ddf-56833979873e Address:fa248718-eaa9-82ca-9ddf-56833979873e}]" === RUN TestRaft_SnapshotRestore_PeerChange --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) raft_test.go:1026: Disabled unreliable test === RUN TestRaft_AutoSnapshot --- SKIP: TestRaft_AutoSnapshot (0.00s) raft_test.go:1132: Disabled unreliable test testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.567Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering follower state: follower="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Follower]" leader= === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.567Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b7e15ad-206e-37d5-ff1d-009d46db69a1 Address:0b7e15ad-206e-37d5-ff1d-009d46db69a1}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.567Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.567Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering follower state: follower="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Follower]" leader= testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [WARN] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering candidate state: node="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Candidate]" term=2 testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: election won: tally=1 testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering leader state: leader="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Leader]" testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.663Z [WARN] server-fa248718-eaa9-82ca-9ddf-56833979873e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.663Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering candidate state: node="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Candidate]" term=3 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.663Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: election won: tally=1 testLoggerAdapter verbose: server-fa248718-eaa9-82ca-9ddf-56833979873e: 2021-10-08T13:01:10.663Z [INFO] server-fa248718-eaa9-82ca-9ddf-56833979873e: entering leader state: leader="Node at fa248718-eaa9-82ca-9ddf-56833979873e [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.738Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.633713813 +0000 UTC m=+10.648396537 (1 nodes), 65.991883ms from start of poll, 66.174539ms from cluster start. Timeout at 2021-10-08 13:01:10.738838249 +0000 UTC m=+10.753520997, 105.12446ms after stability testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.739Z [ERROR] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.739Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: starting snapshot up to: index=12 testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.742Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.742Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.18s) testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.567Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0b7e15ad-206e-37d5-ff1d-009d46db69a1 Address:0b7e15ad-206e-37d5-ff1d-009d46db69a1}]" testing.go:149: cluster: 2021-10-08T13:01:10.567Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.567Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering follower state: follower="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Follower]" leader= testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [WARN] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering candidate state: node="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Candidate]" term=2 testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: election won: tally=1 testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.633Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: entering leader state: leader="Node at 0b7e15ad-206e-37d5-ff1d-009d46db69a1 [Leader]" testing.go:149: cluster: 2021-10-08T13:01:10.738Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.633713813 +0000 UTC m=+10.648396537 (1 nodes), 65.991883ms from start of poll, 66.174539ms from cluster start. Timeout at 2021-10-08 13:01:10.738838249 +0000 UTC m=+10.753520997, 105.12446ms after stability testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.739Z [ERROR] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.739Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:10.739Z [INFO] creating new snapshot: path=/tmp/raft705568149/snapshots/2-12-1633698070739.tmp testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.742Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: compacting logs: from=1 to=2 testing.go:149: server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: 2021-10-08T13:01:10.742Z [INFO] server-0b7e15ad-206e-37d5-ff1d-009d46db69a1: snapshot complete up to: index=12 === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.743Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.744Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testLoggerAdapter verbose: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.743Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: entering follower state: follower="Node at dc475ab0-f100-7c21-a160-d0c607a75f92 [Follower]" leader= testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.744Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering follower state: follower="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Follower]" leader= testLoggerAdapter verbose: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.744Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.744Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.744Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: entering follower state: follower="Node at fae90067-17fe-3e58-6102-a88e67cfeaee [Follower]" leader= testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.816Z [WARN] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering candidate state: node="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Candidate]" term=2 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: election won: tally=2 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering leader state: leader="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Leader]" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: added peer, starting replication: peer=server-dc475ab0-f100-7c21-a160-d0c607a75f92 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: added peer, starting replication: peer=server-fae90067-17fe-3e58-6102-a88e67cfeaee testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.922Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.817282915 +0000 UTC m=+10.831965669 (1 nodes), 72.738777ms from start of poll, 73.970651ms from cluster start. Timeout at 2021-10-08 13:01:10.922398397 +0000 UTC m=+10.937081121, 105.115452ms after stability testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.922Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: starting snapshot up to: index=12 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.925Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: snapshot complete up to: index=12 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.928Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: restored user snapshot: index=1 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.936Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.938Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.938Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.939Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: Installed remote snapshot testLoggerAdapter verbose: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.939Z [WARN] server-fae90067-17fe-3e58-6102-a88e67cfeaee: skipping application of old log: index=22 testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.939Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testLoggerAdapter verbose: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.941Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.941Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: Installed remote snapshot testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.941Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.955Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testLoggerAdapter verbose: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.955Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:10.957Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:10.957Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering follower state: follower="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Follower]" leader= testLoggerAdapter verbose: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:10.958Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testLoggerAdapter verbose: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:10.958Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: entering follower state: follower="Node at 0fe37b27-3492-54c8-ee89-3363551df0d4 [Follower]" leader= testLoggerAdapter verbose: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:10.958Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:10.958Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:10.958Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: entering follower state: follower="Node at e1b69df3-255f-103d-0765-1e4c3a9f3358 [Follower]" leader= testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [WARN] server-f389b94c-5c1c-c7b1-db48-9a3903341467: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering candidate state: node="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Candidate]" term=2 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: election won: tally=2 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering leader state: leader="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Leader]" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: added peer, starting replication: peer=server-0fe37b27-3492-54c8-ee89-3363551df0d4 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: added peer, starting replication: peer=server-e1b69df3-255f-103d-0765-1e4c3a9f3358 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.019Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.019Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.124Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.018999278 +0000 UTC m=+11.033682023 (1 nodes), 60.598353ms from start of poll, 61.318365ms from cluster start. Timeout at 2021-10-08 13:01:11.124105241 +0000 UTC m=+11.138787966, 105.105943ms after stability testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.124Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: starting snapshot up to: index=12 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.127Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: snapshot complete up to: index=12 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: restored user snapshot: index=1 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.136Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.139Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: Installed remote snapshot testLoggerAdapter verbose: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [WARN] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: skipping application of old log: index=22 testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.140Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testLoggerAdapter verbose: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:11.142Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:11.142Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: Installed remote snapshot testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.142Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.156Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testLoggerAdapter verbose: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.156Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testLoggerAdapter verbose: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.159Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testLoggerAdapter verbose: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.159Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: entering follower state: follower="Node at 84638519-68ea-8aa8-e727-ecd10670a327 [Follower]" leader= testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.159Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.159Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering follower state: follower="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Follower]" leader= testLoggerAdapter verbose: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.159Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.159Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.159Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: entering follower state: follower="Node at c836aecd-5a20-7c73-3bab-a59890e29a82 [Follower]" leader= testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [WARN] server-85252766-6881-fd3d-22a2-4fed80a73f9a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering candidate state: node="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Candidate]" term=2 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: election won: tally=2 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering leader state: leader="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Leader]" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: added peer, starting replication: peer=server-84638519-68ea-8aa8-e727-ecd10670a327 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: added peer, starting replication: peer=server-c836aecd-5a20-7c73-3bab-a59890e29a82 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.334Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.22960044 +0000 UTC m=+11.244283160 (1 nodes), 69.782979ms from start of poll, 70.360571ms from cluster start. Timeout at 2021-10-08 13:01:11.334689554 +0000 UTC m=+11.349372274, 105.089114ms after stability testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.335Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: starting snapshot up to: index=12 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.338Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: snapshot complete up to: index=12 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: restored user snapshot: index=1 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.349Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.351Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: Installed remote snapshot testLoggerAdapter verbose: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [WARN] server-84638519-68ea-8aa8-e727-ecd10670a327: skipping application of old log: index=22 testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.353Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testLoggerAdapter verbose: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.354Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.354Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: Installed remote snapshot testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.354Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.369Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testLoggerAdapter verbose: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.369Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.372Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.372Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: entering follower state: follower="Node at 86491b4c-7306-8e09-caef-8d090fa8694a [Follower]" leader= testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.372Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.372Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering follower state: follower="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Follower]" leader= testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.372Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.372Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.372Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering follower state: follower="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Follower]" leader= testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [WARN] server-11c4538e-af0f-d995-6bcb-aae80056bff6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering candidate state: node="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Candidate]" term=2 testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [WARN] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering candidate state: node="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Candidate]" term=2 testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: election won: tally=2 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering leader state: leader="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Leader]" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: added peer, starting replication: peer=server-86491b4c-7306-8e09-caef-8d090fa8694a testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: added peer, starting replication: peer=server-fc5d300a-3288-e147-487e-a5bc7e3fde7f testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [WARN] server-11c4538e-af0f-d995-6bcb-aae80056bff6: rejecting vote request since we have a leader: from=fc5d300a-3288-e147-487e-a5bc7e3fde7f leader=11c4538e-af0f-d995-6bcb-aae80056bff6 testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.425Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering follower state: follower="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Follower]" leader=11c4538e-af0f-d995-6bcb-aae80056bff6 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.530Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.425592406 +0000 UTC m=+11.440275163 (1 nodes), 52.802467ms from start of poll, 53.577938ms from cluster start. Timeout at 2021-10-08 13:01:11.530739703 +0000 UTC m=+11.545422450, 105.147287ms after stability testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.531Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: starting snapshot up to: index=12 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.534Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: snapshot complete up to: index=12 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.537Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.537Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: restored user snapshot: index=1 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.545Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.547Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: Installed remote snapshot testLoggerAdapter verbose: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [WARN] server-86491b4c-7306-8e09-caef-8d090fa8694a: skipping application of old log: index=22 testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.548Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.549Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.549Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: Installed remote snapshot testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.549Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.564Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testLoggerAdapter verbose: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.564Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.567Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.567Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering follower state: follower="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Follower]" leader= testLoggerAdapter verbose: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.567Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testLoggerAdapter verbose: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.567Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: entering follower state: follower="Node at 945a91c8-9b98-3a98-c433-b4d11fd55c8b [Follower]" leader= testLoggerAdapter verbose: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.567Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.568Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.568Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: entering follower state: follower="Node at 44749693-5bf7-79f3-50fb-46453d4bf6f2 [Follower]" leader= testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [WARN] server-6fd00b20-aa38-00ab-b892-5f89ef873786: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering candidate state: node="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Candidate]" term=2 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: election won: tally=2 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering leader state: leader="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Leader]" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: added peer, starting replication: peer=server-945a91c8-9b98-3a98-c433-b4d11fd55c8b testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: added peer, starting replication: peer=server-44749693-5bf7-79f3-50fb-46453d4bf6f2 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.723Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.618663243 +0000 UTC m=+11.633345964 (1 nodes), 50.57675ms from start of poll, 51.273952ms from cluster start. Timeout at 2021-10-08 13:01:11.723728523 +0000 UTC m=+11.738411245, 105.065281ms after stability testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.724Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: starting snapshot up to: index=12 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.726Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: snapshot complete up to: index=12 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: restored user snapshot: index=1 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.735Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.738Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: Installed remote snapshot testLoggerAdapter verbose: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [WARN] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: skipping application of old log: index=22 testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.738Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testLoggerAdapter verbose: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.740Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.740Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: Installed remote snapshot testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.740Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.754Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.754Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testLoggerAdapter verbose: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.804Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to heartbeat to: peer=44749693-5bf7-79f3-50fb-46453d4bf6f2 error="command timed out" testLoggerAdapter verbose: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:11.806Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.806Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testLoggerAdapter verbose: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:11.806Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: entering follower state: follower="Node at ba03c920-0ff4-f781-3a3a-af7f29f98479 [Follower]" leader= testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.806Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering follower state: follower="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Follower]" leader= testLoggerAdapter verbose: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.806Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.806Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.806Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: entering follower state: follower="Node at c0d067dc-9e0e-77cc-4829-afe1a155a282 [Follower]" leader= testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.875Z [WARN] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.875Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering candidate state: node="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Candidate]" term=2 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: election won: tally=2 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering leader state: leader="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Leader]" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: added peer, starting replication: peer=server-ba03c920-0ff4-f781-3a3a-af7f29f98479 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: added peer, starting replication: peer=server-c0d067dc-9e0e-77cc-4829-afe1a155a282 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:11.981Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.876153107 +0000 UTC m=+11.890835837 (1 nodes), 69.264372ms from start of poll, 69.725402ms from cluster start. Timeout at 2021-10-08 13:01:11.981251236 +0000 UTC m=+11.995933958, 105.098121ms after stability testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.981Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: starting snapshot up to: index=12 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.987Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: snapshot complete up to: index=12 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: restored user snapshot: index=1 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.991Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.996Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.999Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.999Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.999Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: Installed remote snapshot testLoggerAdapter verbose: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:12.000Z [WARN] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: skipping application of old log: index=22 testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.000Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testLoggerAdapter verbose: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:12.002Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:12.002Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: Installed remote snapshot testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.002Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.016Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testLoggerAdapter verbose: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.016Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" --- PASS: TestRaft_UserRestore (1.28s) testing.go:149: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.743Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.744Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testing.go:149: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.743Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: entering follower state: follower="Node at dc475ab0-f100-7c21-a160-d0c607a75f92 [Follower]" leader= testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.744Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering follower state: follower="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Follower]" leader= testing.go:149: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.744Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dc475ab0-f100-7c21-a160-d0c607a75f92 Address:dc475ab0-f100-7c21-a160-d0c607a75f92} {Suffrage:Voter ID:server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 Address:b7ef5cbc-6841-61bb-f472-eefbf4ec5be7} {Suffrage:Voter ID:server-fae90067-17fe-3e58-6102-a88e67cfeaee Address:fae90067-17fe-3e58-6102-a88e67cfeaee}]" testing.go:149: cluster: 2021-10-08T13:01:10.744Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.744Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: entering follower state: follower="Node at fae90067-17fe-3e58-6102-a88e67cfeaee [Follower]" leader= testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.816Z [WARN] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering candidate state: node="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Candidate]" term=2 testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: election won: tally=2 testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: entering leader state: leader="Node at b7ef5cbc-6841-61bb-f472-eefbf4ec5be7 [Leader]" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: added peer, starting replication: peer=server-dc475ab0-f100-7c21-a160-d0c607a75f92 testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: added peer, starting replication: peer=server-fae90067-17fe-3e58-6102-a88e67cfeaee testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.817Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testing.go:149: cluster: 2021-10-08T13:01:10.922Z [INFO] stable state for Leader reached at 2021-10-08 13:01:10.817282915 +0000 UTC m=+10.831965669 (1 nodes), 72.738777ms from start of poll, 73.970651ms from cluster start. Timeout at 2021-10-08 13:01:10.922398397 +0000 UTC m=+10.937081121, 105.115452ms after stability testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.922Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:10.922Z [INFO] creating new snapshot: path=/tmp/raft527312793/snapshots/2-12-1633698070922.tmp testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.925Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:10.926Z [INFO] creating new snapshot: path=/tmp/raft527312793/snapshots/2-23-1633698070926.tmp testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.928Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: copied to local snapshot: bytes=71 testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: restored user snapshot: index=1 testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.929Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.936Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:10.936Z [INFO] creating new snapshot: path=/tmp/raft797090611/snapshots/2-23-1633698070936.tmp testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.938Z [ERROR] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: failed to get log: index=23 error="log not found" testing.go:149: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.938Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: copied to local snapshot: bytes=71 testing.go:149: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.939Z [INFO] server-fae90067-17fe-3e58-6102-a88e67cfeaee: Installed remote snapshot testing.go:149: server-fae90067-17fe-3e58-6102-a88e67cfeaee: 2021-10-08T13:01:10.939Z [WARN] server-fae90067-17fe-3e58-6102-a88e67cfeaee: skipping application of old log: index=22 testing.go:153: 2021-10-08T13:01:10.939Z [INFO] creating new snapshot: path=/tmp/raft581080719/snapshots/2-23-1633698070939.tmp testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.939Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testing.go:149: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.941Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: copied to local snapshot: bytes=71 testing.go:149: server-dc475ab0-f100-7c21-a160-d0c607a75f92: 2021-10-08T13:01:10.941Z [INFO] server-dc475ab0-f100-7c21-a160-d0c607a75f92: Installed remote snapshot testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.941Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: pipelining replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.955Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-fae90067-17fe-3e58-6102-a88e67cfeaee fae90067-17fe-3e58-6102-a88e67cfeaee}" testing.go:149: server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: 2021-10-08T13:01:10.955Z [INFO] server-b7ef5cbc-6841-61bb-f472-eefbf4ec5be7: aborting pipeline replication: peer="{Voter server-dc475ab0-f100-7c21-a160-d0c607a75f92 dc475ab0-f100-7c21-a160-d0c607a75f92}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:10.957Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:10.957Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering follower state: follower="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Follower]" leader= testing.go:149: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:10.958Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testing.go:149: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:10.958Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: entering follower state: follower="Node at 0fe37b27-3492-54c8-ee89-3363551df0d4 [Follower]" leader= testing.go:149: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:10.958Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-f389b94c-5c1c-c7b1-db48-9a3903341467 Address:f389b94c-5c1c-c7b1-db48-9a3903341467} {Suffrage:Voter ID:server-0fe37b27-3492-54c8-ee89-3363551df0d4 Address:0fe37b27-3492-54c8-ee89-3363551df0d4} {Suffrage:Voter ID:server-e1b69df3-255f-103d-0765-1e4c3a9f3358 Address:e1b69df3-255f-103d-0765-1e4c3a9f3358}]" testing.go:149: cluster: 2021-10-08T13:01:10.958Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:10.958Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: entering follower state: follower="Node at e1b69df3-255f-103d-0765-1e4c3a9f3358 [Follower]" leader= testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [WARN] server-f389b94c-5c1c-c7b1-db48-9a3903341467: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering candidate state: node="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Candidate]" term=2 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: election won: tally=2 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: entering leader state: leader="Node at f389b94c-5c1c-c7b1-db48-9a3903341467 [Leader]" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: added peer, starting replication: peer=server-0fe37b27-3492-54c8-ee89-3363551df0d4 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.018Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: added peer, starting replication: peer=server-e1b69df3-255f-103d-0765-1e4c3a9f3358 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.019Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.019Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testing.go:149: cluster: 2021-10-08T13:01:11.124Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.018999278 +0000 UTC m=+11.033682023 (1 nodes), 60.598353ms from start of poll, 61.318365ms from cluster start. Timeout at 2021-10-08 13:01:11.124105241 +0000 UTC m=+11.138787966, 105.105943ms after stability testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.124Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:11.124Z [INFO] creating new snapshot: path=/tmp/raft747363293/snapshots/2-12-1633698071124.tmp testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.127Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:11.128Z [INFO] creating new snapshot: path=/tmp/raft747363293/snapshots/2-23-1633698071128.tmp testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: copied to local snapshot: bytes=71 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: restored user snapshot: index=1 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.131Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.136Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.137Z [INFO] creating new snapshot: path=/tmp/raft229372001/snapshots/2-23-1633698071137.tmp testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.139Z [ERROR] server-f389b94c-5c1c-c7b1-db48-9a3903341467: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.139Z [INFO] creating new snapshot: path=/tmp/raft666566167/snapshots/2-23-1633698071139.tmp testing.go:149: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: copied to local snapshot: bytes=71 testing.go:149: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [INFO] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: Installed remote snapshot testing.go:149: server-e1b69df3-255f-103d-0765-1e4c3a9f3358: 2021-10-08T13:01:11.140Z [WARN] server-e1b69df3-255f-103d-0765-1e4c3a9f3358: skipping application of old log: index=22 testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.140Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testing.go:149: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:11.142Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: copied to local snapshot: bytes=71 testing.go:149: server-0fe37b27-3492-54c8-ee89-3363551df0d4: 2021-10-08T13:01:11.142Z [INFO] server-0fe37b27-3492-54c8-ee89-3363551df0d4: Installed remote snapshot testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.142Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: pipelining replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.156Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-e1b69df3-255f-103d-0765-1e4c3a9f3358 e1b69df3-255f-103d-0765-1e4c3a9f3358}" testing.go:149: server-f389b94c-5c1c-c7b1-db48-9a3903341467: 2021-10-08T13:01:11.156Z [INFO] server-f389b94c-5c1c-c7b1-db48-9a3903341467: aborting pipeline replication: peer="{Voter server-0fe37b27-3492-54c8-ee89-3363551df0d4 0fe37b27-3492-54c8-ee89-3363551df0d4}" testing.go:149: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.159Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testing.go:149: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.159Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: entering follower state: follower="Node at 84638519-68ea-8aa8-e727-ecd10670a327 [Follower]" leader= testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.159Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.159Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering follower state: follower="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Follower]" leader= testing.go:149: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.159Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-84638519-68ea-8aa8-e727-ecd10670a327 Address:84638519-68ea-8aa8-e727-ecd10670a327} {Suffrage:Voter ID:server-85252766-6881-fd3d-22a2-4fed80a73f9a Address:85252766-6881-fd3d-22a2-4fed80a73f9a} {Suffrage:Voter ID:server-c836aecd-5a20-7c73-3bab-a59890e29a82 Address:c836aecd-5a20-7c73-3bab-a59890e29a82}]" testing.go:149: cluster: 2021-10-08T13:01:11.159Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.159Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: entering follower state: follower="Node at c836aecd-5a20-7c73-3bab-a59890e29a82 [Follower]" leader= testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [WARN] server-85252766-6881-fd3d-22a2-4fed80a73f9a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering candidate state: node="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Candidate]" term=2 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: election won: tally=2 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: entering leader state: leader="Node at 85252766-6881-fd3d-22a2-4fed80a73f9a [Leader]" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: added peer, starting replication: peer=server-84638519-68ea-8aa8-e727-ecd10670a327 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: added peer, starting replication: peer=server-c836aecd-5a20-7c73-3bab-a59890e29a82 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.229Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testing.go:149: cluster: 2021-10-08T13:01:11.334Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.22960044 +0000 UTC m=+11.244283160 (1 nodes), 69.782979ms from start of poll, 70.360571ms from cluster start. Timeout at 2021-10-08 13:01:11.334689554 +0000 UTC m=+11.349372274, 105.089114ms after stability testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.335Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:11.335Z [INFO] creating new snapshot: path=/tmp/raft744950565/snapshots/2-12-1633698071335.tmp testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.338Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:11.339Z [INFO] creating new snapshot: path=/tmp/raft744950565/snapshots/2-23-1633698071339.tmp testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: copied to local snapshot: bytes=71 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: restored user snapshot: index=1 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.341Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.342Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.349Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.350Z [INFO] creating new snapshot: path=/tmp/raft086707003/snapshots/2-23-1633698071350.tmp testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.351Z [ERROR] server-85252766-6881-fd3d-22a2-4fed80a73f9a: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.351Z [INFO] creating new snapshot: path=/tmp/raft352425631/snapshots/2-23-1633698071351.tmp testing.go:149: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: copied to local snapshot: bytes=71 testing.go:149: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [INFO] server-84638519-68ea-8aa8-e727-ecd10670a327: Installed remote snapshot testing.go:149: server-84638519-68ea-8aa8-e727-ecd10670a327: 2021-10-08T13:01:11.353Z [WARN] server-84638519-68ea-8aa8-e727-ecd10670a327: skipping application of old log: index=22 testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.353Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testing.go:149: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.354Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: copied to local snapshot: bytes=71 testing.go:149: server-c836aecd-5a20-7c73-3bab-a59890e29a82: 2021-10-08T13:01:11.354Z [INFO] server-c836aecd-5a20-7c73-3bab-a59890e29a82: Installed remote snapshot testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.354Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: pipelining replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.369Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-c836aecd-5a20-7c73-3bab-a59890e29a82 c836aecd-5a20-7c73-3bab-a59890e29a82}" testing.go:149: server-85252766-6881-fd3d-22a2-4fed80a73f9a: 2021-10-08T13:01:11.369Z [INFO] server-85252766-6881-fd3d-22a2-4fed80a73f9a: aborting pipeline replication: peer="{Voter server-84638519-68ea-8aa8-e727-ecd10670a327 84638519-68ea-8aa8-e727-ecd10670a327}" testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.372Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.372Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: entering follower state: follower="Node at 86491b4c-7306-8e09-caef-8d090fa8694a [Follower]" leader= testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.372Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.372Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering follower state: follower="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Follower]" leader= testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.372Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-86491b4c-7306-8e09-caef-8d090fa8694a Address:86491b4c-7306-8e09-caef-8d090fa8694a} {Suffrage:Voter ID:server-11c4538e-af0f-d995-6bcb-aae80056bff6 Address:11c4538e-af0f-d995-6bcb-aae80056bff6} {Suffrage:Voter ID:server-fc5d300a-3288-e147-487e-a5bc7e3fde7f Address:fc5d300a-3288-e147-487e-a5bc7e3fde7f}]" testing.go:149: cluster: 2021-10-08T13:01:11.372Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.372Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering follower state: follower="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Follower]" leader= testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [WARN] server-11c4538e-af0f-d995-6bcb-aae80056bff6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering candidate state: node="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Candidate]" term=2 testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [WARN] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering candidate state: node="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Candidate]" term=2 testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: duplicate requestVote for same term: term=2 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: election won: tally=2 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: entering leader state: leader="Node at 11c4538e-af0f-d995-6bcb-aae80056bff6 [Leader]" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: added peer, starting replication: peer=server-86491b4c-7306-8e09-caef-8d090fa8694a testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: added peer, starting replication: peer=server-fc5d300a-3288-e147-487e-a5bc7e3fde7f testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [WARN] server-11c4538e-af0f-d995-6bcb-aae80056bff6: rejecting vote request since we have a leader: from=fc5d300a-3288-e147-487e-a5bc7e3fde7f leader=11c4538e-af0f-d995-6bcb-aae80056bff6 testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.425Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: duplicate requestVote for same term: term=2 testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.425Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: entering follower state: follower="Node at fc5d300a-3288-e147-487e-a5bc7e3fde7f [Follower]" leader=11c4538e-af0f-d995-6bcb-aae80056bff6 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.425Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testing.go:149: cluster: 2021-10-08T13:01:11.530Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.425592406 +0000 UTC m=+11.440275163 (1 nodes), 52.802467ms from start of poll, 53.577938ms from cluster start. Timeout at 2021-10-08 13:01:11.530739703 +0000 UTC m=+11.545422450, 105.147287ms after stability testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.531Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:11.531Z [INFO] creating new snapshot: path=/tmp/raft580869187/snapshots/2-12-1633698071531.tmp testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.534Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:11.535Z [INFO] creating new snapshot: path=/tmp/raft580869187/snapshots/2-113-1633698071535.tmp testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.537Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: copied to local snapshot: bytes=71 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.537Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: restored user snapshot: index=1 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.538Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.545Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.546Z [INFO] creating new snapshot: path=/tmp/raft188327465/snapshots/2-113-1633698071546.tmp testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.547Z [ERROR] server-11c4538e-af0f-d995-6bcb-aae80056bff6: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.547Z [INFO] creating new snapshot: path=/tmp/raft007844205/snapshots/2-113-1633698071547.tmp testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: copied to local snapshot: bytes=71 testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [INFO] server-86491b4c-7306-8e09-caef-8d090fa8694a: Installed remote snapshot testing.go:149: server-86491b4c-7306-8e09-caef-8d090fa8694a: 2021-10-08T13:01:11.548Z [WARN] server-86491b4c-7306-8e09-caef-8d090fa8694a: skipping application of old log: index=22 testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.548Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.549Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: copied to local snapshot: bytes=71 testing.go:149: server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: 2021-10-08T13:01:11.549Z [INFO] server-fc5d300a-3288-e147-487e-a5bc7e3fde7f: Installed remote snapshot testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.549Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: pipelining replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.564Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-fc5d300a-3288-e147-487e-a5bc7e3fde7f fc5d300a-3288-e147-487e-a5bc7e3fde7f}" testing.go:149: server-11c4538e-af0f-d995-6bcb-aae80056bff6: 2021-10-08T13:01:11.564Z [INFO] server-11c4538e-af0f-d995-6bcb-aae80056bff6: aborting pipeline replication: peer="{Voter server-86491b4c-7306-8e09-caef-8d090fa8694a 86491b4c-7306-8e09-caef-8d090fa8694a}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.567Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.567Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering follower state: follower="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Follower]" leader= testing.go:149: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.567Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testing.go:149: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.567Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: entering follower state: follower="Node at 945a91c8-9b98-3a98-c433-b4d11fd55c8b [Follower]" leader= testing.go:149: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.567Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6fd00b20-aa38-00ab-b892-5f89ef873786 Address:6fd00b20-aa38-00ab-b892-5f89ef873786} {Suffrage:Voter ID:server-945a91c8-9b98-3a98-c433-b4d11fd55c8b Address:945a91c8-9b98-3a98-c433-b4d11fd55c8b} {Suffrage:Voter ID:server-44749693-5bf7-79f3-50fb-46453d4bf6f2 Address:44749693-5bf7-79f3-50fb-46453d4bf6f2}]" testing.go:149: cluster: 2021-10-08T13:01:11.568Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.568Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: entering follower state: follower="Node at 44749693-5bf7-79f3-50fb-46453d4bf6f2 [Follower]" leader= testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [WARN] server-6fd00b20-aa38-00ab-b892-5f89ef873786: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering candidate state: node="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Candidate]" term=2 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: election won: tally=2 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: entering leader state: leader="Node at 6fd00b20-aa38-00ab-b892-5f89ef873786 [Leader]" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: added peer, starting replication: peer=server-945a91c8-9b98-3a98-c433-b4d11fd55c8b testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: added peer, starting replication: peer=server-44749693-5bf7-79f3-50fb-46453d4bf6f2 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.618Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testing.go:149: cluster: 2021-10-08T13:01:11.723Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.618663243 +0000 UTC m=+11.633345964 (1 nodes), 50.57675ms from start of poll, 51.273952ms from cluster start. Timeout at 2021-10-08 13:01:11.723728523 +0000 UTC m=+11.738411245, 105.065281ms after stability testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.724Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:11.724Z [INFO] creating new snapshot: path=/tmp/raft691830823/snapshots/2-12-1633698071724.tmp testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.726Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:11.727Z [INFO] creating new snapshot: path=/tmp/raft691830823/snapshots/2-1013-1633698071727.tmp testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: copied to local snapshot: bytes=71 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: restored user snapshot: index=1 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.729Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.735Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.736Z [INFO] creating new snapshot: path=/tmp/raft160566001/snapshots/2-1013-1633698071736.tmp testing.go:149: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: copied to local snapshot: bytes=71 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.738Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to get log: index=23 error="log not found" testing.go:149: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [INFO] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: Installed remote snapshot testing.go:149: server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: 2021-10-08T13:01:11.738Z [WARN] server-945a91c8-9b98-3a98-c433-b4d11fd55c8b: skipping application of old log: index=22 testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.738Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testing.go:153: 2021-10-08T13:01:11.738Z [INFO] creating new snapshot: path=/tmp/raft244092491/snapshots/2-1013-1633698071738.tmp testing.go:149: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.740Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: copied to local snapshot: bytes=71 testing.go:149: server-44749693-5bf7-79f3-50fb-46453d4bf6f2: 2021-10-08T13:01:11.740Z [INFO] server-44749693-5bf7-79f3-50fb-46453d4bf6f2: Installed remote snapshot testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.740Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: pipelining replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.754Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-945a91c8-9b98-3a98-c433-b4d11fd55c8b 945a91c8-9b98-3a98-c433-b4d11fd55c8b}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.754Z [INFO] server-6fd00b20-aa38-00ab-b892-5f89ef873786: aborting pipeline replication: peer="{Voter server-44749693-5bf7-79f3-50fb-46453d4bf6f2 44749693-5bf7-79f3-50fb-46453d4bf6f2}" testing.go:149: server-6fd00b20-aa38-00ab-b892-5f89ef873786: 2021-10-08T13:01:11.804Z [ERROR] server-6fd00b20-aa38-00ab-b892-5f89ef873786: failed to heartbeat to: peer=44749693-5bf7-79f3-50fb-46453d4bf6f2 error="command timed out" testing.go:149: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:11.806Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.806Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testing.go:149: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:11.806Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: entering follower state: follower="Node at ba03c920-0ff4-f781-3a3a-af7f29f98479 [Follower]" leader= testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.806Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering follower state: follower="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Follower]" leader= testing.go:149: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.806Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-ba03c920-0ff4-f781-3a3a-af7f29f98479 Address:ba03c920-0ff4-f781-3a3a-af7f29f98479} {Suffrage:Voter ID:server-342583e5-51d2-499d-dc1f-e9278d6d9e88 Address:342583e5-51d2-499d-dc1f-e9278d6d9e88} {Suffrage:Voter ID:server-c0d067dc-9e0e-77cc-4829-afe1a155a282 Address:c0d067dc-9e0e-77cc-4829-afe1a155a282}]" testing.go:149: cluster: 2021-10-08T13:01:11.806Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.806Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: entering follower state: follower="Node at c0d067dc-9e0e-77cc-4829-afe1a155a282 [Follower]" leader= testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.875Z [WARN] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.875Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering candidate state: node="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Candidate]" term=2 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: election won: tally=2 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: entering leader state: leader="Node at 342583e5-51d2-499d-dc1f-e9278d6d9e88 [Leader]" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: added peer, starting replication: peer=server-ba03c920-0ff4-f781-3a3a-af7f29f98479 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: added peer, starting replication: peer=server-c0d067dc-9e0e-77cc-4829-afe1a155a282 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.876Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testing.go:149: cluster: 2021-10-08T13:01:11.981Z [INFO] stable state for Leader reached at 2021-10-08 13:01:11.876153107 +0000 UTC m=+11.890835837 (1 nodes), 69.264372ms from start of poll, 69.725402ms from cluster start. Timeout at 2021-10-08 13:01:11.981251236 +0000 UTC m=+11.995933958, 105.098121ms after stability testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.981Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: starting snapshot up to: index=12 testing.go:153: 2021-10-08T13:01:11.981Z [INFO] creating new snapshot: path=/tmp/raft321613487/snapshots/2-12-1633698071981.tmp testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.987Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: snapshot complete up to: index=12 testing.go:153: 2021-10-08T13:01:11.988Z [INFO] creating new snapshot: path=/tmp/raft321613487/snapshots/2-10013-1633698071988.tmp testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: copied to local snapshot: bytes=71 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: restored user snapshot: index=1 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.991Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.990Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.996Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.997Z [INFO] creating new snapshot: path=/tmp/raft449410233/snapshots/2-10013-1633698071997.tmp testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:11.999Z [ERROR] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-08T13:01:11.999Z [INFO] creating new snapshot: path=/tmp/raft819043509/snapshots/2-10013-1633698071999.tmp testing.go:149: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.999Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: copied to local snapshot: bytes=71 testing.go:149: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:11.999Z [INFO] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: Installed remote snapshot testing.go:149: server-c0d067dc-9e0e-77cc-4829-afe1a155a282: 2021-10-08T13:01:12.000Z [WARN] server-c0d067dc-9e0e-77cc-4829-afe1a155a282: skipping application of old log: index=22 testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.000Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testing.go:149: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:12.002Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: copied to local snapshot: bytes=71 testing.go:149: server-ba03c920-0ff4-f781-3a3a-af7f29f98479: 2021-10-08T13:01:12.002Z [INFO] server-ba03c920-0ff4-f781-3a3a-af7f29f98479: Installed remote snapshot testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.002Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: pipelining replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.016Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-c0d067dc-9e0e-77cc-4829-afe1a155a282 c0d067dc-9e0e-77cc-4829-afe1a155a282}" testing.go:149: server-342583e5-51d2-499d-dc1f-e9278d6d9e88: 2021-10-08T13:01:12.016Z [INFO] server-342583e5-51d2-499d-dc1f-e9278d6d9e88: aborting pipeline replication: peer="{Voter server-ba03c920-0ff4-f781-3a3a-af7f29f98479 ba03c920-0ff4-f781-3a3a-af7f29f98479}" === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.018Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.019Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: entering follower state: follower="Node at 29973e3d-785a-a3ed-6cd1-23dd8987270b [Follower]" leader= testLoggerAdapter verbose: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.019Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testLoggerAdapter verbose: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.019Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: entering follower state: follower="Node at 14240858-71c9-9f43-4ecd-232e1f493d0f [Follower]" leader= testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.019Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.019Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.019Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering follower state: follower="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Follower]" leader= testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.092Z [WARN] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.092Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering candidate state: node="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Candidate]" term=2 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: election won: tally=2 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering leader state: leader="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Leader]" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: added peer, starting replication: peer=server-29973e3d-785a-a3ed-6cd1-23dd8987270b testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: added peer, starting replication: peer=server-14240858-71c9-9f43-4ecd-232e1f493d0f testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-14240858-71c9-9f43-4ecd-232e1f493d0f 14240858-71c9-9f43-4ecd-232e1f493d0f}" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.198Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.093155765 +0000 UTC m=+12.107838514 (2 nodes), 73.749033ms from start of poll, 74.413663ms from cluster start. Timeout at 2021-10-08 13:01:12.198221644 +0000 UTC m=+12.212904370, 105.065856ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.198Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.303Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.198365691 +0000 UTC m=+12.213048408 (1 nodes), 924ns from start of poll, 179.623557ms from cluster start. Timeout at 2021-10-08 13:01:12.303438047 +0000 UTC m=+12.318120778, 105.07237ms after stability testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to pipeline appendEntries: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" error="append pipeline closed" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to appendEntries to: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" error="failed to connect to peer: 29973e3d-785a-a3ed-6cd1-23dd8987270b" testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.305Z [ERROR] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.305Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: starting snapshot up to: index=101 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.307Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to heartbeat to: peer=29973e3d-785a-a3ed-6cd1-23dd8987270b error="failed to connect to peer: 29973e3d-785a-a3ed-6cd1-23dd8987270b" testLoggerAdapter verbose: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.308Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.308Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: snapshot complete up to: index=101 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.308Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: starting snapshot up to: index=102 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.310Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.310Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: snapshot complete up to: index=102 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.313Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: Installed remote snapshot testLoggerAdapter verbose: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [WARN] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: skipping application of old log: index=102 testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.317Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.320Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-14240858-71c9-9f43-4ecd-232e1f493d0f 14240858-71c9-9f43-4ecd-232e1f493d0f}" testLoggerAdapter verbose: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.320Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" --- PASS: TestRaft_SendSnapshotFollower (0.30s) testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.018Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.019Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: entering follower state: follower="Node at 29973e3d-785a-a3ed-6cd1-23dd8987270b [Follower]" leader= testing.go:149: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.019Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testing.go:149: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.019Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: entering follower state: follower="Node at 14240858-71c9-9f43-4ecd-232e1f493d0f [Follower]" leader= testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.019Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-29973e3d-785a-a3ed-6cd1-23dd8987270b Address:29973e3d-785a-a3ed-6cd1-23dd8987270b} {Suffrage:Voter ID:server-14240858-71c9-9f43-4ecd-232e1f493d0f Address:14240858-71c9-9f43-4ecd-232e1f493d0f} {Suffrage:Voter ID:server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3 Address:8f1234d0-76e4-d086-26ce-1e81f6bcbae3}]" testing.go:149: cluster: 2021-10-08T13:01:12.019Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.019Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering follower state: follower="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Follower]" leader= testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.092Z [WARN] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.092Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering candidate state: node="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Candidate]" term=2 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: election won: tally=2 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: entering leader state: leader="Node at 8f1234d0-76e4-d086-26ce-1e81f6bcbae3 [Leader]" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: added peer, starting replication: peer=server-29973e3d-785a-a3ed-6cd1-23dd8987270b testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: added peer, starting replication: peer=server-14240858-71c9-9f43-4ecd-232e1f493d0f testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-14240858-71c9-9f43-4ecd-232e1f493d0f 14240858-71c9-9f43-4ecd-232e1f493d0f}" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.093Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testing.go:149: cluster: 2021-10-08T13:01:12.198Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.093155765 +0000 UTC m=+12.107838514 (2 nodes), 73.749033ms from start of poll, 74.413663ms from cluster start. Timeout at 2021-10-08 13:01:12.198221644 +0000 UTC m=+12.212904370, 105.065856ms after stability testing.go:149: cluster: 2021-10-08T13:01:12.198Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:12.303Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.198365691 +0000 UTC m=+12.213048408 (1 nodes), 924ns from start of poll, 179.623557ms from cluster start. Timeout at 2021-10-08 13:01:12.303438047 +0000 UTC m=+12.318120778, 105.07237ms after stability testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to pipeline appendEntries: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" error="append pipeline closed" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.303Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to appendEntries to: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" error="failed to connect to peer: 29973e3d-785a-a3ed-6cd1-23dd8987270b" raft_test.go:1320: [INFO] Finished apply without behind follower testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.305Z [ERROR] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.305Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: starting snapshot up to: index=101 testing.go:153: 2021-10-08T13:01:12.305Z [INFO] creating new snapshot: path=/tmp/raft112172285/snapshots/2-101-1633698072305.tmp testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.307Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to heartbeat to: peer=29973e3d-785a-a3ed-6cd1-23dd8987270b error="failed to connect to peer: 29973e3d-785a-a3ed-6cd1-23dd8987270b" testing.go:149: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.308Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: compacting logs: from=1 to=92 testing.go:149: server-14240858-71c9-9f43-4ecd-232e1f493d0f: 2021-10-08T13:01:12.308Z [INFO] server-14240858-71c9-9f43-4ecd-232e1f493d0f: snapshot complete up to: index=101 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.308Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: starting snapshot up to: index=102 testing.go:153: 2021-10-08T13:01:12.308Z [INFO] creating new snapshot: path=/tmp/raft758209079/snapshots/2-102-1633698072308.tmp testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.310Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: compacting logs: from=1 to=92 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.310Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: snapshot complete up to: index=102 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.313Z [ERROR] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: failed to get log: index=2 error="log not found" testing.go:153: 2021-10-08T13:01:12.314Z [INFO] creating new snapshot: path=/tmp/raft017744211/snapshots/2-102-1633698072314.tmp testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: copied to local snapshot: bytes=693 testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [INFO] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: Installed remote snapshot testing.go:149: server-29973e3d-785a-a3ed-6cd1-23dd8987270b: 2021-10-08T13:01:12.317Z [WARN] server-29973e3d-785a-a3ed-6cd1-23dd8987270b: skipping application of old log: index=102 testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.317Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: pipelining replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.320Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-14240858-71c9-9f43-4ecd-232e1f493d0f 14240858-71c9-9f43-4ecd-232e1f493d0f}" testing.go:149: server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: 2021-10-08T13:01:12.320Z [INFO] server-8f1234d0-76e4-d086-26ce-1e81f6bcbae3: aborting pipeline replication: peer="{Voter server-29973e3d-785a-a3ed-6cd1-23dd8987270b 29973e3d-785a-a3ed-6cd1-23dd8987270b}" === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.322Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testLoggerAdapter verbose: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.322Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testLoggerAdapter verbose: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.322Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: entering follower state: follower="Node at 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a [Follower]" leader= testLoggerAdapter verbose: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.322Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: entering follower state: follower="Node at 8914c6fc-cc5c-0d5d-555c-96f1346dcf23 [Follower]" leader= testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.323Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.323Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.323Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering follower state: follower="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Follower]" leader= testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [WARN] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering candidate state: node="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Candidate]" term=2 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: election won: tally=2 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering leader state: leader="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Leader]" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: added peer, starting replication: peer=server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: added peer, starting replication: peer=server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.394Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 8914c6fc-cc5c-0d5d-555c-96f1346dcf23}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.499Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.393814071 +0000 UTC m=+12.408496802 (2 nodes), 70.708579ms from start of poll, 71.11754ms from cluster start. Timeout at 2021-10-08 13:01:12.498940945 +0000 UTC m=+12.513623668, 105.126866ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.499Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.604Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.499101824 +0000 UTC m=+12.513784545 (1 nodes), 834ns from start of poll, 176.405283ms from cluster start. Timeout at 2021-10-08 13:01:12.604176405 +0000 UTC m=+12.618859149, 105.074604ms after stability testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to pipeline appendEntries: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" error="append pipeline closed" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to appendEntries to: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" error="failed to connect to peer: 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a" testLoggerAdapter verbose: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.605Z [ERROR] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.605Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: starting snapshot up to: index=98 testLoggerAdapter verbose: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.609Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.609Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: snapshot complete up to: index=98 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.609Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: starting snapshot up to: index=102 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.609Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to heartbeat to: peer=65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a error="failed to connect to peer: 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.611Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.611Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: snapshot complete up to: index=102 testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.614Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.616Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.617Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: Installed remote snapshot testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.617Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.623Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 8914c6fc-cc5c-0d5d-555c-96f1346dcf23}" testLoggerAdapter verbose: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.623Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) testing.go:149: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.322Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testing.go:149: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.322Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testing.go:149: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.322Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: entering follower state: follower="Node at 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a [Follower]" leader= testing.go:149: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.322Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: entering follower state: follower="Node at 8914c6fc-cc5c-0d5d-555c-96f1346dcf23 [Follower]" leader= testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.323Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a Address:65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a} {Suffrage:Voter ID:server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 Address:8914c6fc-cc5c-0d5d-555c-96f1346dcf23} {Suffrage:Voter ID:server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 Address:144c1d7f-bf31-7b9e-dbac-af7d5f0e4359}]" testing.go:149: cluster: 2021-10-08T13:01:12.323Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.323Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering follower state: follower="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Follower]" leader= testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [WARN] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering candidate state: node="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Candidate]" term=2 testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: election won: tally=2 testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: entering leader state: leader="Node at 144c1d7f-bf31-7b9e-dbac-af7d5f0e4359 [Leader]" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: added peer, starting replication: peer=server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: added peer, starting replication: peer=server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.393Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.394Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 8914c6fc-cc5c-0d5d-555c-96f1346dcf23}" testing.go:149: cluster: 2021-10-08T13:01:12.499Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.393814071 +0000 UTC m=+12.408496802 (2 nodes), 70.708579ms from start of poll, 71.11754ms from cluster start. Timeout at 2021-10-08 13:01:12.498940945 +0000 UTC m=+12.513623668, 105.126866ms after stability testing.go:149: cluster: 2021-10-08T13:01:12.499Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:12.604Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.499101824 +0000 UTC m=+12.513784545 (1 nodes), 834ns from start of poll, 176.405283ms from cluster start. Timeout at 2021-10-08 13:01:12.604176405 +0000 UTC m=+12.618859149, 105.074604ms after stability testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to pipeline appendEntries: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" error="append pipeline closed" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.604Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to appendEntries to: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" error="failed to connect to peer: 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a" raft_test.go:1362: [INFO] Finished apply without behind follower testing.go:149: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.605Z [ERROR] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.605Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: starting snapshot up to: index=98 testing.go:153: 2021-10-08T13:01:12.605Z [INFO] creating new snapshot: path=/tmp/raft439044955/snapshots/2-98-1633698072605.tmp testing.go:149: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.609Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: compacting logs: from=1 to=92 testing.go:149: server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: 2021-10-08T13:01:12.609Z [INFO] server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23: snapshot complete up to: index=98 testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.609Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: starting snapshot up to: index=102 testing.go:153: 2021-10-08T13:01:12.609Z [INFO] creating new snapshot: path=/tmp/raft319897157/snapshots/2-102-1633698072609.tmp testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.609Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to heartbeat to: peer=65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a error="failed to connect to peer: 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.611Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: compacting logs: from=1 to=92 testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.611Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: snapshot complete up to: index=102 raft_test.go:1383: [INFO] Finished apply without behind follower testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.614Z [ERROR] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: failed to get log: index=2 error="log not found" testing.go:153: 2021-10-08T13:01:12.615Z [INFO] creating new snapshot: path=/tmp/raft061770113/snapshots/2-102-1633698072615.tmp testing.go:149: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.616Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: copied to local snapshot: bytes=693 testing.go:149: server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: 2021-10-08T13:01:12.617Z [INFO] server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a: Installed remote snapshot testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.617Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: pipelining replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.623Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-8914c6fc-cc5c-0d5d-555c-96f1346dcf23 8914c6fc-cc5c-0d5d-555c-96f1346dcf23}" testing.go:149: server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: 2021-10-08T13:01:12.623Z [INFO] server-144c1d7f-bf31-7b9e-dbac-af7d5f0e4359: aborting pipeline replication: peer="{Voter server-65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a 65c5cc51-f7e0-83d9-c13a-fb10ebe0e24a}" === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:12.626Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:12.626Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering follower state: follower="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Follower]" leader= testLoggerAdapter verbose: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:12.626Z [INFO] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testLoggerAdapter verbose: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:12.626Z [INFO] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: entering follower state: follower="Node at 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b [Follower]" leader= testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.626Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.627Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.627Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering follower state: follower="Node at e89e1406-b148-794f-efeb-855b9204707b [Follower]" leader= testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering candidate state: node="Node at e89e1406-b148-794f-efeb-855b9204707b [Candidate]" term=2 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: election won: tally=2 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering leader state: leader="Node at e89e1406-b148-794f-efeb-855b9204707b [Leader]" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.790Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.685432148 +0000 UTC m=+12.700114877 (1 nodes), 57.994046ms from start of poll, 59.577169ms from cluster start. Timeout at 2021-10-08 13:01:12.790546372 +0000 UTC m=+12.805229093, 105.114216ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.795Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:12.900Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.795827465 +0000 UTC m=+12.810510184 (2 nodes), 801ns from start of poll, 169.972476ms from cluster start. Timeout at 2021-10-08 13:01:12.90089611 +0000 UTC m=+12.915578845, 105.068661ms after stability testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: updating configuration: command=RemoveServer server-id=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 server-addr= servers="[{Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: removed peer, stopping replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 last-index=3 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.009Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: heartbeat timeout reached, starting election: last-leader=e89e1406-b148-794f-efeb-855b9204707b testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.009Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering candidate state: node="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Candidate]" term=3 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testLoggerAdapter verbose: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:13.009Z [WARN] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: updating configuration: command=AddStaging server-id=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 server-addr=be47a506-e684-46a6-e2c1-e6b0f05d1231 servers="[{Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b} {Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231}]" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [ERROR] server-e89e1406-b148-794f-efeb-855b9204707b: peer has newer term, stopping replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering follower state: follower="Node at e89e1406-b148-794f-efeb-855b9204707b [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.009Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.093Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: Election timeout reached, restarting election testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.093Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering candidate state: node="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Candidate]" term=4 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.093Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: rejecting vote request since our last index is greater: candidate=be47a506-e684-46a6-e2c1-e6b0f05d1231 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:13.093Z [WARN] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering candidate state: node="Node at e89e1406-b148-794f-efeb-855b9204707b [Candidate]" term=5 testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.100Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering follower state: follower="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Follower]" leader= testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: election won: tally=2 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering leader state: leader="Node at e89e1406-b148-794f-efeb-855b9204707b [Leader]" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testLoggerAdapter verbose: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.100Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: appendEntries rejected, sending older logs: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" next=4 testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.205Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.100683505 +0000 UTC m=+13.115366269 (1 nodes), 91.094884ms from start of poll, 474.828561ms from cluster start. Timeout at 2021-10-08 13:01:13.205766383 +0000 UTC m=+13.220449118, 105.082849ms after stability testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.205Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testLoggerAdapter verbose: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.205Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" --- PASS: TestRaft_ReJoinFollower (0.58s) testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:12.626Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testing.go:149: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:12.626Z [INFO] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:12.626Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering follower state: follower="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Follower]" leader= testing.go:149: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:12.626Z [INFO] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: entering follower state: follower="Node at 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b [Follower]" leader= testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.626Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231} {Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testing.go:149: cluster: 2021-10-08T13:01:12.627Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.627Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering follower state: follower="Node at e89e1406-b148-794f-efeb-855b9204707b [Follower]" leader= testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering candidate state: node="Node at e89e1406-b148-794f-efeb-855b9204707b [Candidate]" term=2 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: election won: tally=2 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering leader state: leader="Node at e89e1406-b148-794f-efeb-855b9204707b [Leader]" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.685Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testing.go:149: cluster: 2021-10-08T13:01:12.790Z [INFO] stable state for Leader reached at 2021-10-08 13:01:12.685432148 +0000 UTC m=+12.700114877 (1 nodes), 57.994046ms from start of poll, 59.577169ms from cluster start. Timeout at 2021-10-08 13:01:12.790546372 +0000 UTC m=+12.805229093, 105.114216ms after stability testing.go:149: cluster: 2021-10-08T13:01:12.795Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:12.900Z [INFO] stable state for Follower reached at 2021-10-08 13:01:12.795827465 +0000 UTC m=+12.810510184 (2 nodes), 801ns from start of poll, 169.972476ms from cluster start. Timeout at 2021-10-08 13:01:12.90089611 +0000 UTC m=+12.915578845, 105.068661ms after stability testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: updating configuration: command=RemoveServer server-id=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 server-addr= servers="[{Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b}]" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: removed peer, stopping replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 last-index=3 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:12.901Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.009Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: heartbeat timeout reached, starting election: last-leader=e89e1406-b148-794f-efeb-855b9204707b testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.009Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering candidate state: node="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Candidate]" term=3 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testing.go:149: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:13.009Z [WARN] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: updating configuration: command=AddStaging server-id=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 server-addr=be47a506-e684-46a6-e2c1-e6b0f05d1231 servers="[{Suffrage:Voter ID:server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b Address:813d1fb1-0a11-d2f2-eeae-2dbaf06d695b} {Suffrage:Voter ID:server-e89e1406-b148-794f-efeb-855b9204707b Address:e89e1406-b148-794f-efeb-855b9204707b} {Suffrage:Voter ID:server-be47a506-e684-46a6-e2c1-e6b0f05d1231 Address:be47a506-e684-46a6-e2c1-e6b0f05d1231}]" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [ERROR] server-e89e1406-b148-794f-efeb-855b9204707b: peer has newer term, stopping replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering follower state: follower="Node at e89e1406-b148-794f-efeb-855b9204707b [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:13.009Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.009Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.093Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: Election timeout reached, restarting election testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.093Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering candidate state: node="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Candidate]" term=4 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.093Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: rejecting vote request since our last index is greater: candidate=be47a506-e684-46a6-e2c1-e6b0f05d1231 last-index=4 last-candidate-index=3 testing.go:149: server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: 2021-10-08T13:01:13.093Z [WARN] server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b: rejecting vote request since we have a leader: from=be47a506-e684-46a6-e2c1-e6b0f05d1231 leader=e89e1406-b148-794f-efeb-855b9204707b testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering candidate state: node="Node at e89e1406-b148-794f-efeb-855b9204707b [Candidate]" term=5 testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.100Z [INFO] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: entering follower state: follower="Node at be47a506-e684-46a6-e2c1-e6b0f05d1231 [Follower]" leader= testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: election won: tally=2 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: entering leader state: leader="Node at e89e1406-b148-794f-efeb-855b9204707b [Leader]" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: added peer, starting replication: peer=server-be47a506-e684-46a6-e2c1-e6b0f05d1231 testing.go:149: server-be47a506-e684-46a6-e2c1-e6b0f05d1231: 2021-10-08T13:01:13.100Z [WARN] server-be47a506-e684-46a6-e2c1-e6b0f05d1231: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [WARN] server-e89e1406-b148-794f-efeb-855b9204707b: appendEntries rejected, sending older logs: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" next=4 testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.100Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: pipelining replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testing.go:149: cluster: 2021-10-08T13:01:13.205Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.100683505 +0000 UTC m=+13.115366269 (1 nodes), 91.094884ms from start of poll, 474.828561ms from cluster start. Timeout at 2021-10-08 13:01:13.205766383 +0000 UTC m=+13.220449118, 105.082849ms after stability testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.205Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-be47a506-e684-46a6-e2c1-e6b0f05d1231 be47a506-e684-46a6-e2c1-e6b0f05d1231}" testing.go:149: server-e89e1406-b148-794f-efeb-855b9204707b: 2021-10-08T13:01:13.205Z [INFO] server-e89e1406-b148-794f-efeb-855b9204707b: aborting pipeline replication: peer="{Voter server-813d1fb1-0a11-d2f2-eeae-2dbaf06d695b 813d1fb1-0a11-d2f2-eeae-2dbaf06d695b}" === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.207Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 Address:b2577e8f-d3f6-b697-e1b5-7c77491907e9} {Suffrage:Voter ID:server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 Address:4549d5a8-adc4-e7e7-c564-bf60e88f7f59}]" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.207Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering follower state: follower="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Follower]" leader= testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.207Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 Address:b2577e8f-d3f6-b697-e1b5-7c77491907e9} {Suffrage:Voter ID:server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 Address:4549d5a8-adc4-e7e7-c564-bf60e88f7f59}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.207Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.207Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering follower state: follower="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Follower]" leader= testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=2 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: election won: tally=2 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering leader state: leader="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Leader]" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: added peer, starting replication: peer=server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: pipelining replication: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.396Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.291596401 +0000 UTC m=+13.306279152 (1 nodes), 83.918883ms from start of poll, 84.245528ms from cluster start. Timeout at 2021-10-08 13:01:13.39666811 +0000 UTC m=+13.411350843, 105.071691ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.401Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.507Z [INFO] stable state for Follower reached at 2021-10-08 13:01:13.40202032 +0000 UTC m=+13.416703100 (1 nodes), 751ns from start of poll, 194.669476ms from cluster start. Timeout at 2021-10-08 13:01:13.507123579 +0000 UTC m=+13.521806299, 105.103199ms after stability testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.511Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to pipeline appendEntries: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="append pipeline closed" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.511Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: aborting pipeline replication: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.513Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.519Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.533Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.538Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.550Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to contact: server-id=server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 time=50.173361ms testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering follower state: follower="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.555Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.557Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=3 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: heartbeat timeout reached, starting election: last-leader=b2577e8f-d3f6-b697-e1b5-7c77491907e9 testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=3 testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=4 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: Election timeout reached, restarting election testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=4 testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.748Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.643277486 +0000 UTC m=+13.657960245 (0 nodes), 87.711666ms from start of poll, 435.926621ms from cluster start. Timeout at 2021-10-08 13:01:13.748402473 +0000 UTC m=+13.763085221, 105.124976ms after stability testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=5 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: Election timeout reached, restarting election testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=5 testLoggerAdapter verbose: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=6 testLoggerAdapter verbose: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" --- PASS: TestRaft_LeaderLeaseExpire (0.65s) testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.207Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 Address:b2577e8f-d3f6-b697-e1b5-7c77491907e9} {Suffrage:Voter ID:server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 Address:4549d5a8-adc4-e7e7-c564-bf60e88f7f59}]" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.207Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering follower state: follower="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Follower]" leader= testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.207Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 Address:b2577e8f-d3f6-b697-e1b5-7c77491907e9} {Suffrage:Voter ID:server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 Address:4549d5a8-adc4-e7e7-c564-bf60e88f7f59}]" testing.go:149: cluster: 2021-10-08T13:01:13.207Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.207Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering follower state: follower="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Follower]" leader= testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=2 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: election won: tally=2 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering leader state: leader="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Leader]" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: added peer, starting replication: peer=server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.291Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: pipelining replication: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" testing.go:149: cluster: 2021-10-08T13:01:13.396Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.291596401 +0000 UTC m=+13.306279152 (1 nodes), 83.918883ms from start of poll, 84.245528ms from cluster start. Timeout at 2021-10-08 13:01:13.39666811 +0000 UTC m=+13.411350843, 105.071691ms after stability testing.go:149: cluster: 2021-10-08T13:01:13.401Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:13.507Z [INFO] stable state for Follower reached at 2021-10-08 13:01:13.40202032 +0000 UTC m=+13.416703100 (1 nodes), 751ns from start of poll, 194.669476ms from cluster start. Timeout at 2021-10-08 13:01:13.507123579 +0000 UTC m=+13.521806299, 105.103199ms after stability raft_test.go:1492: [INFO] Disconnecting Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Follower] testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.511Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to pipeline appendEntries: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="append pipeline closed" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.511Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: aborting pipeline replication: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.513Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.519Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.533Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.538Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.550Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to heartbeat to: peer=4549d5a8-adc4-e7e7-c564-bf60e88f7f59 error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to contact: server-id=server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 time=50.173361ms testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to contact quorum of nodes, stepping down testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.555Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering follower state: follower="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:13.555Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.557Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to appendEntries to: peer="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=3 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.627Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: heartbeat timeout reached, starting election: last-leader=b2577e8f-d3f6-b697-e1b5-7c77491907e9 testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=3 testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.643Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=4 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.678Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: Election timeout reached, restarting election testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=4 testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.736Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testing.go:149: cluster: 2021-10-08T13:01:13.748Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.643277486 +0000 UTC m=+13.657960245 (0 nodes), 87.711666ms from start of poll, 435.926621ms from cluster start. Timeout at 2021-10-08 13:01:13.748402473 +0000 UTC m=+13.763085221, 105.124976ms after stability testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=5 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.768Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [WARN] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: Election timeout reached, restarting election testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [INFO] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: entering candidate state: node="Node at 4549d5a8-adc4-e7e7-c564-bf60e88f7f59 [Candidate]" term=5 testing.go:149: server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: 2021-10-08T13:01:13.815Z [ERROR] server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59: failed to make requestVote RPC: target="{Voter server-b2577e8f-d3f6-b697-e1b5-7c77491907e9 b2577e8f-d3f6-b697-e1b5-7c77491907e9}" error="failed to connect to peer: b2577e8f-d3f6-b697-e1b5-7c77491907e9" testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [WARN] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: Election timeout reached, restarting election testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [INFO] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: entering candidate state: node="Node at b2577e8f-d3f6-b697-e1b5-7c77491907e9 [Candidate]" term=6 testing.go:149: server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: 2021-10-08T13:01:13.835Z [ERROR] server-b2577e8f-d3f6-b697-e1b5-7c77491907e9: failed to make requestVote RPC: target="{Voter server-4549d5a8-adc4-e7e7-c564-bf60e88f7f59 4549d5a8-adc4-e7e7-c564-bf60e88f7f59}" error="failed to connect to peer: 4549d5a8-adc4-e7e7-c564-bf60e88f7f59" === RUN TestRaft_Barrier testLoggerAdapter verbose: server-0706e247-3078-2aa3-c469-8fc1fa521b0f: 2021-10-08T13:01:13.855Z [INFO] server-0706e247-3078-2aa3-c469-8fc1fa521b0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.855Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testLoggerAdapter verbose: server-0706e247-3078-2aa3-c469-8fc1fa521b0f: 2021-10-08T13:01:13.855Z [INFO] server-0706e247-3078-2aa3-c469-8fc1fa521b0f: entering follower state: follower="Node at 0706e247-3078-2aa3-c469-8fc1fa521b0f [Follower]" leader= testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.856Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering follower state: follower="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Follower]" leader= testLoggerAdapter verbose: server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: 2021-10-08T13:01:13.856Z [INFO] server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:13.856Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: 2021-10-08T13:01:13.856Z [INFO] server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: entering follower state: follower="Node at 74b9b03e-d03a-5d91-c2e1-78a9082940d7 [Follower]" leader= testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [WARN] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering candidate state: node="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Candidate]" term=2 testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: election won: tally=2 testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering leader state: leader="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Leader]" testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: added peer, starting replication: peer=server-0706e247-3078-2aa3-c469-8fc1fa521b0f testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: added peer, starting replication: peer=server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: pipelining replication: peer="{Voter server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 74b9b03e-d03a-5d91-c2e1-78a9082940d7}" testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: pipelining replication: peer="{Voter server-0706e247-3078-2aa3-c469-8fc1fa521b0f 0706e247-3078-2aa3-c469-8fc1fa521b0f}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.032Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.927078607 +0000 UTC m=+13.941761329 (1 nodes), 70.858553ms from start of poll, 71.592719ms from cluster start. Timeout at 2021-10-08 13:01:14.032154649 +0000 UTC m=+14.046837395, 105.076066ms after stability testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:14.045Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: aborting pipeline replication: peer="{Voter server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 74b9b03e-d03a-5d91-c2e1-78a9082940d7}" testLoggerAdapter verbose: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:14.045Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: aborting pipeline replication: peer="{Voter server-0706e247-3078-2aa3-c469-8fc1fa521b0f 0706e247-3078-2aa3-c469-8fc1fa521b0f}" --- PASS: TestRaft_Barrier (0.19s) testing.go:149: server-0706e247-3078-2aa3-c469-8fc1fa521b0f: 2021-10-08T13:01:13.855Z [INFO] server-0706e247-3078-2aa3-c469-8fc1fa521b0f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testing.go:149: server-0706e247-3078-2aa3-c469-8fc1fa521b0f: 2021-10-08T13:01:13.855Z [INFO] server-0706e247-3078-2aa3-c469-8fc1fa521b0f: entering follower state: follower="Node at 0706e247-3078-2aa3-c469-8fc1fa521b0f [Follower]" leader= testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.855Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.856Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering follower state: follower="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Follower]" leader= testing.go:149: server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: 2021-10-08T13:01:13.856Z [INFO] server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0706e247-3078-2aa3-c469-8fc1fa521b0f Address:0706e247-3078-2aa3-c469-8fc1fa521b0f} {Suffrage:Voter ID:server-9ae5f6af-b84b-f893-4337-1cee11b438d0 Address:9ae5f6af-b84b-f893-4337-1cee11b438d0} {Suffrage:Voter ID:server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 Address:74b9b03e-d03a-5d91-c2e1-78a9082940d7}]" testing.go:149: cluster: 2021-10-08T13:01:13.856Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: 2021-10-08T13:01:13.856Z [INFO] server-74b9b03e-d03a-5d91-c2e1-78a9082940d7: entering follower state: follower="Node at 74b9b03e-d03a-5d91-c2e1-78a9082940d7 [Follower]" leader= testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [WARN] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering candidate state: node="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Candidate]" term=2 testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: election won: tally=2 testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.926Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: entering leader state: leader="Node at 9ae5f6af-b84b-f893-4337-1cee11b438d0 [Leader]" testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: added peer, starting replication: peer=server-0706e247-3078-2aa3-c469-8fc1fa521b0f testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: added peer, starting replication: peer=server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: pipelining replication: peer="{Voter server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 74b9b03e-d03a-5d91-c2e1-78a9082940d7}" testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:13.927Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: pipelining replication: peer="{Voter server-0706e247-3078-2aa3-c469-8fc1fa521b0f 0706e247-3078-2aa3-c469-8fc1fa521b0f}" testing.go:149: cluster: 2021-10-08T13:01:14.032Z [INFO] stable state for Leader reached at 2021-10-08 13:01:13.927078607 +0000 UTC m=+13.941761329 (1 nodes), 70.858553ms from start of poll, 71.592719ms from cluster start. Timeout at 2021-10-08 13:01:14.032154649 +0000 UTC m=+14.046837395, 105.076066ms after stability testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:14.045Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: aborting pipeline replication: peer="{Voter server-74b9b03e-d03a-5d91-c2e1-78a9082940d7 74b9b03e-d03a-5d91-c2e1-78a9082940d7}" testing.go:149: server-9ae5f6af-b84b-f893-4337-1cee11b438d0: 2021-10-08T13:01:14.045Z [INFO] server-9ae5f6af-b84b-f893-4337-1cee11b438d0: aborting pipeline replication: peer="{Voter server-0706e247-3078-2aa3-c469-8fc1fa521b0f 0706e247-3078-2aa3-c469-8fc1fa521b0f}" === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-31394264-e801-7b7d-7072-92e8267025d2: 2021-10-08T13:01:14.046Z [INFO] server-31394264-e801-7b7d-7072-92e8267025d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.046Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testLoggerAdapter verbose: server-31394264-e801-7b7d-7072-92e8267025d2: 2021-10-08T13:01:14.046Z [INFO] server-31394264-e801-7b7d-7072-92e8267025d2: entering follower state: follower="Node at 31394264-e801-7b7d-7072-92e8267025d2 [Follower]" leader= testLoggerAdapter verbose: server-a2580272-674a-91f8-b993-4db3381b1aae: 2021-10-08T13:01:14.046Z [INFO] server-a2580272-674a-91f8-b993-4db3381b1aae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.046Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.046Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering follower state: follower="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Follower]" leader= testLoggerAdapter verbose: server-a2580272-674a-91f8-b993-4db3381b1aae: 2021-10-08T13:01:14.047Z [INFO] server-a2580272-674a-91f8-b993-4db3381b1aae: entering follower state: follower="Node at a2580272-674a-91f8-b993-4db3381b1aae [Follower]" leader= testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [WARN] server-a9397635-e470-69ef-4e0b-9b5be0704464: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering candidate state: node="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Candidate]" term=2 testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: election won: tally=2 testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering leader state: leader="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Leader]" testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: added peer, starting replication: peer=server-31394264-e801-7b7d-7072-92e8267025d2 testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: added peer, starting replication: peer=server-a2580272-674a-91f8-b993-4db3381b1aae testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: pipelining replication: peer="{Voter server-31394264-e801-7b7d-7072-92e8267025d2 31394264-e801-7b7d-7072-92e8267025d2}" testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: pipelining replication: peer="{Voter server-a2580272-674a-91f8-b993-4db3381b1aae a2580272-674a-91f8-b993-4db3381b1aae}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.210Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.104864158 +0000 UTC m=+14.119546910 (1 nodes), 57.85825ms from start of poll, 58.281098ms from cluster start. Timeout at 2021-10-08 13:01:14.209989898 +0000 UTC m=+14.224672618, 105.125708ms after stability testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.210Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: aborting pipeline replication: peer="{Voter server-a2580272-674a-91f8-b993-4db3381b1aae a2580272-674a-91f8-b993-4db3381b1aae}" testLoggerAdapter verbose: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.210Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: aborting pipeline replication: peer="{Voter server-31394264-e801-7b7d-7072-92e8267025d2 31394264-e801-7b7d-7072-92e8267025d2}" --- PASS: TestRaft_VerifyLeader (0.17s) testing.go:149: server-31394264-e801-7b7d-7072-92e8267025d2: 2021-10-08T13:01:14.046Z [INFO] server-31394264-e801-7b7d-7072-92e8267025d2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.046Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testing.go:149: server-31394264-e801-7b7d-7072-92e8267025d2: 2021-10-08T13:01:14.046Z [INFO] server-31394264-e801-7b7d-7072-92e8267025d2: entering follower state: follower="Node at 31394264-e801-7b7d-7072-92e8267025d2 [Follower]" leader= testing.go:149: server-a2580272-674a-91f8-b993-4db3381b1aae: 2021-10-08T13:01:14.046Z [INFO] server-a2580272-674a-91f8-b993-4db3381b1aae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31394264-e801-7b7d-7072-92e8267025d2 Address:31394264-e801-7b7d-7072-92e8267025d2} {Suffrage:Voter ID:server-a9397635-e470-69ef-4e0b-9b5be0704464 Address:a9397635-e470-69ef-4e0b-9b5be0704464} {Suffrage:Voter ID:server-a2580272-674a-91f8-b993-4db3381b1aae Address:a2580272-674a-91f8-b993-4db3381b1aae}]" testing.go:149: cluster: 2021-10-08T13:01:14.046Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.046Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering follower state: follower="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Follower]" leader= testing.go:149: server-a2580272-674a-91f8-b993-4db3381b1aae: 2021-10-08T13:01:14.047Z [INFO] server-a2580272-674a-91f8-b993-4db3381b1aae: entering follower state: follower="Node at a2580272-674a-91f8-b993-4db3381b1aae [Follower]" leader= testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [WARN] server-a9397635-e470-69ef-4e0b-9b5be0704464: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering candidate state: node="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Candidate]" term=2 testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: election won: tally=2 testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: entering leader state: leader="Node at a9397635-e470-69ef-4e0b-9b5be0704464 [Leader]" testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: added peer, starting replication: peer=server-31394264-e801-7b7d-7072-92e8267025d2 testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: added peer, starting replication: peer=server-a2580272-674a-91f8-b993-4db3381b1aae testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: pipelining replication: peer="{Voter server-31394264-e801-7b7d-7072-92e8267025d2 31394264-e801-7b7d-7072-92e8267025d2}" testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.104Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: pipelining replication: peer="{Voter server-a2580272-674a-91f8-b993-4db3381b1aae a2580272-674a-91f8-b993-4db3381b1aae}" testing.go:149: cluster: 2021-10-08T13:01:14.210Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.104864158 +0000 UTC m=+14.119546910 (1 nodes), 57.85825ms from start of poll, 58.281098ms from cluster start. Timeout at 2021-10-08 13:01:14.209989898 +0000 UTC m=+14.224672618, 105.125708ms after stability testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.210Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: aborting pipeline replication: peer="{Voter server-a2580272-674a-91f8-b993-4db3381b1aae a2580272-674a-91f8-b993-4db3381b1aae}" testing.go:149: server-a9397635-e470-69ef-4e0b-9b5be0704464: 2021-10-08T13:01:14.210Z [INFO] server-a9397635-e470-69ef-4e0b-9b5be0704464: aborting pipeline replication: peer="{Voter server-31394264-e801-7b7d-7072-92e8267025d2 31394264-e801-7b7d-7072-92e8267025d2}" === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.211Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35def6dc-776b-0946-7e7a-fcc3aa20f97a Address:35def6dc-776b-0946-7e7a-fcc3aa20f97a}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.211Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.211Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering follower state: follower="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Follower]" leader= testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [WARN] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering candidate state: node="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Candidate]" term=2 testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: election won: tally=1 testLoggerAdapter verbose: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering leader state: leader="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.398Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.293266215 +0000 UTC m=+14.307948936 (1 nodes), 81.549874ms from start of poll, 81.817204ms from cluster start. Timeout at 2021-10-08 13:01:14.398281643 +0000 UTC m=+14.412964367, 105.015431ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.19s) testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.211Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-35def6dc-776b-0946-7e7a-fcc3aa20f97a Address:35def6dc-776b-0946-7e7a-fcc3aa20f97a}]" testing.go:149: cluster: 2021-10-08T13:01:14.211Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.211Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering follower state: follower="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Follower]" leader= testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [WARN] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering candidate state: node="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Candidate]" term=2 testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: election won: tally=1 testing.go:149: server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: 2021-10-08T13:01:14.293Z [INFO] server-35def6dc-776b-0946-7e7a-fcc3aa20f97a: entering leader state: leader="Node at 35def6dc-776b-0946-7e7a-fcc3aa20f97a [Leader]" testing.go:149: cluster: 2021-10-08T13:01:14.398Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.293266215 +0000 UTC m=+14.307948936 (1 nodes), 81.549874ms from start of poll, 81.817204ms from cluster start. Timeout at 2021-10-08 13:01:14.398281643 +0000 UTC m=+14.412964367, 105.015431ms after stability === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.399Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4 Address:4f66fd96-f1dd-e318-a68f-66ce709ec7b4} {Suffrage:Voter ID:server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 Address:d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}]" testLoggerAdapter verbose: server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: 2021-10-08T13:01:14.399Z [INFO] server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4 Address:4f66fd96-f1dd-e318-a68f-66ce709ec7b4} {Suffrage:Voter ID:server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 Address:d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}]" testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.399Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering follower state: follower="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.399Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: 2021-10-08T13:01:14.399Z [INFO] server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: entering follower state: follower="Node at d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 [Follower]" leader= testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [WARN] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering candidate state: node="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Candidate]" term=2 testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: election won: tally=2 testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering leader state: leader="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Leader]" testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: added peer, starting replication: peer=server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: pipelining replication: peer="{Voter server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.575Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.470557556 +0000 UTC m=+14.485240361 (1 nodes), 70.965124ms from start of poll, 71.290202ms from cluster start. Timeout at 2021-10-08 13:01:14.575644256 +0000 UTC m=+14.590326989, 105.086628ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.575Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.680Z [INFO] stable state for Follower reached at 2021-10-08 13:01:14.575799347 +0000 UTC m=+14.590482075 (1 nodes), 697ns from start of poll, 176.531916ms from cluster start. Timeout at 2021-10-08 13:01:14.680868972 +0000 UTC m=+14.695551691, 105.069616ms after stability testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.680Z [WARN] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: new leader elected, stepping down testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.681Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering follower state: follower="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Follower]" leader= testLoggerAdapter verbose: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.681Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: aborting pipeline replication: peer="{Voter server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}" --- PASS: TestRaft_VerifyLeader_Fail (0.28s) testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.399Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4 Address:4f66fd96-f1dd-e318-a68f-66ce709ec7b4} {Suffrage:Voter ID:server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 Address:d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}]" testing.go:149: server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: 2021-10-08T13:01:14.399Z [INFO] server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4 Address:4f66fd96-f1dd-e318-a68f-66ce709ec7b4} {Suffrage:Voter ID:server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 Address:d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}]" testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.399Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering follower state: follower="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:14.399Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: 2021-10-08T13:01:14.399Z [INFO] server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67: entering follower state: follower="Node at d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 [Follower]" leader= testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [WARN] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering candidate state: node="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Candidate]" term=2 testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: election won: tally=2 testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering leader state: leader="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Leader]" testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: added peer, starting replication: peer=server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.470Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: pipelining replication: peer="{Voter server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}" testing.go:149: cluster: 2021-10-08T13:01:14.575Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.470557556 +0000 UTC m=+14.485240361 (1 nodes), 70.965124ms from start of poll, 71.290202ms from cluster start. Timeout at 2021-10-08 13:01:14.575644256 +0000 UTC m=+14.590326989, 105.086628ms after stability testing.go:149: cluster: 2021-10-08T13:01:14.575Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:14.680Z [INFO] stable state for Follower reached at 2021-10-08 13:01:14.575799347 +0000 UTC m=+14.590482075 (1 nodes), 697ns from start of poll, 176.531916ms from cluster start. Timeout at 2021-10-08 13:01:14.680868972 +0000 UTC m=+14.695551691, 105.069616ms after stability testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.680Z [WARN] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: new leader elected, stepping down testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.681Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: entering follower state: follower="Node at 4f66fd96-f1dd-e318-a68f-66ce709ec7b4 [Follower]" leader= testing.go:149: server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: 2021-10-08T13:01:14.681Z [INFO] server-4f66fd96-f1dd-e318-a68f-66ce709ec7b4: aborting pipeline replication: peer="{Voter server-d97ae7ad-f836-d6ff-e7cb-9bab9718ac67 d97ae7ad-f836-d6ff-e7cb-9bab9718ac67}" === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-79ea62c3-77be-3c82-5df3-5e81f77d3028: 2021-10-08T13:01:14.682Z [INFO] server-79ea62c3-77be-3c82-5df3-5e81f77d3028: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testLoggerAdapter verbose: server-79ea62c3-77be-3c82-5df3-5e81f77d3028: 2021-10-08T13:01:14.682Z [INFO] server-79ea62c3-77be-3c82-5df3-5e81f77d3028: entering follower state: follower="Node at 79ea62c3-77be-3c82-5df3-5e81f77d3028 [Follower]" leader= testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.682Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.682Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering follower state: follower="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Follower]" leader= testLoggerAdapter verbose: server-77331087-c731-b0ff-acbe-92cb7841695e: 2021-10-08T13:01:14.683Z [INFO] server-77331087-c731-b0ff-acbe-92cb7841695e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.683Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-77331087-c731-b0ff-acbe-92cb7841695e: 2021-10-08T13:01:14.683Z [INFO] server-77331087-c731-b0ff-acbe-92cb7841695e: entering follower state: follower="Node at 77331087-c731-b0ff-acbe-92cb7841695e [Follower]" leader= testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [WARN] server-90f9cada-4406-a978-7703-2ec2690b3c22: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering candidate state: node="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Candidate]" term=2 testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: election won: tally=2 testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering leader state: leader="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Leader]" testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: added peer, starting replication: peer=server-79ea62c3-77be-3c82-5df3-5e81f77d3028 testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: added peer, starting replication: peer=server-77331087-c731-b0ff-acbe-92cb7841695e testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: pipelining replication: peer="{Voter server-77331087-c731-b0ff-acbe-92cb7841695e 77331087-c731-b0ff-acbe-92cb7841695e}" testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: pipelining replication: peer="{Voter server-79ea62c3-77be-3c82-5df3-5e81f77d3028 79ea62c3-77be-3c82-5df3-5e81f77d3028}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.839Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.734422639 +0000 UTC m=+14.749105360 (1 nodes), 51.303661ms from start of poll, 51.891577ms from cluster start. Timeout at 2021-10-08 13:01:14.839541915 +0000 UTC m=+14.854224640, 105.11928ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.844Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:14.950Z [INFO] stable state for Follower reached at 2021-10-08 13:01:14.844839775 +0000 UTC m=+14.859522496 (2 nodes), 915ns from start of poll, 162.308713ms from cluster start. Timeout at 2021-10-08 13:01:14.949901419 +0000 UTC m=+14.964584146, 105.06165ms after stability testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [ERROR] server-90f9cada-4406-a978-7703-2ec2690b3c22: failed to heartbeat to: peer=79ea62c3-77be-3c82-5df3-5e81f77d3028 error="failed to connect to peer: 79ea62c3-77be-3c82-5df3-5e81f77d3028" testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: aborting pipeline replication: peer="{Voter server-77331087-c731-b0ff-acbe-92cb7841695e 77331087-c731-b0ff-acbe-92cb7841695e}" testLoggerAdapter verbose: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: aborting pipeline replication: peer="{Voter server-79ea62c3-77be-3c82-5df3-5e81f77d3028 79ea62c3-77be-3c82-5df3-5e81f77d3028}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) testing.go:149: server-79ea62c3-77be-3c82-5df3-5e81f77d3028: 2021-10-08T13:01:14.682Z [INFO] server-79ea62c3-77be-3c82-5df3-5e81f77d3028: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testing.go:149: server-79ea62c3-77be-3c82-5df3-5e81f77d3028: 2021-10-08T13:01:14.682Z [INFO] server-79ea62c3-77be-3c82-5df3-5e81f77d3028: entering follower state: follower="Node at 79ea62c3-77be-3c82-5df3-5e81f77d3028 [Follower]" leader= testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.682Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.682Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering follower state: follower="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Follower]" leader= testing.go:149: server-77331087-c731-b0ff-acbe-92cb7841695e: 2021-10-08T13:01:14.683Z [INFO] server-77331087-c731-b0ff-acbe-92cb7841695e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79ea62c3-77be-3c82-5df3-5e81f77d3028 Address:79ea62c3-77be-3c82-5df3-5e81f77d3028} {Suffrage:Voter ID:server-90f9cada-4406-a978-7703-2ec2690b3c22 Address:90f9cada-4406-a978-7703-2ec2690b3c22} {Suffrage:Voter ID:server-77331087-c731-b0ff-acbe-92cb7841695e Address:77331087-c731-b0ff-acbe-92cb7841695e}]" testing.go:149: cluster: 2021-10-08T13:01:14.683Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-77331087-c731-b0ff-acbe-92cb7841695e: 2021-10-08T13:01:14.683Z [INFO] server-77331087-c731-b0ff-acbe-92cb7841695e: entering follower state: follower="Node at 77331087-c731-b0ff-acbe-92cb7841695e [Follower]" leader= testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [WARN] server-90f9cada-4406-a978-7703-2ec2690b3c22: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering candidate state: node="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Candidate]" term=2 testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: election won: tally=2 testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: entering leader state: leader="Node at 90f9cada-4406-a978-7703-2ec2690b3c22 [Leader]" testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: added peer, starting replication: peer=server-79ea62c3-77be-3c82-5df3-5e81f77d3028 testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: added peer, starting replication: peer=server-77331087-c731-b0ff-acbe-92cb7841695e testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: pipelining replication: peer="{Voter server-77331087-c731-b0ff-acbe-92cb7841695e 77331087-c731-b0ff-acbe-92cb7841695e}" testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.734Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: pipelining replication: peer="{Voter server-79ea62c3-77be-3c82-5df3-5e81f77d3028 79ea62c3-77be-3c82-5df3-5e81f77d3028}" testing.go:149: cluster: 2021-10-08T13:01:14.839Z [INFO] stable state for Leader reached at 2021-10-08 13:01:14.734422639 +0000 UTC m=+14.749105360 (1 nodes), 51.303661ms from start of poll, 51.891577ms from cluster start. Timeout at 2021-10-08 13:01:14.839541915 +0000 UTC m=+14.854224640, 105.11928ms after stability testing.go:149: cluster: 2021-10-08T13:01:14.844Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:14.950Z [INFO] stable state for Follower reached at 2021-10-08 13:01:14.844839775 +0000 UTC m=+14.859522496 (2 nodes), 915ns from start of poll, 162.308713ms from cluster start. Timeout at 2021-10-08 13:01:14.949901419 +0000 UTC m=+14.964584146, 105.06165ms after stability raft_test.go:1647: [INFO] Disconnecting Node at 79ea62c3-77be-3c82-5df3-5e81f77d3028 [Follower] testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [ERROR] server-90f9cada-4406-a978-7703-2ec2690b3c22: failed to heartbeat to: peer=79ea62c3-77be-3c82-5df3-5e81f77d3028 error="failed to connect to peer: 79ea62c3-77be-3c82-5df3-5e81f77d3028" testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: aborting pipeline replication: peer="{Voter server-77331087-c731-b0ff-acbe-92cb7841695e 77331087-c731-b0ff-acbe-92cb7841695e}" testing.go:149: server-90f9cada-4406-a978-7703-2ec2690b3c22: 2021-10-08T13:01:14.950Z [INFO] server-90f9cada-4406-a978-7703-2ec2690b3c22: aborting pipeline replication: peer="{Voter server-79ea62c3-77be-3c82-5df3-5e81f77d3028 79ea62c3-77be-3c82-5df3-5e81f77d3028}" === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-b9e03c73-1a26-2c99-0894-c73344e94a9f: 2021-10-08T13:01:14.952Z [INFO] server-b9e03c73-1a26-2c99-0894-c73344e94a9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9e03c73-1a26-2c99-0894-c73344e94a9f Address:b9e03c73-1a26-2c99-0894-c73344e94a9f}]" testLoggerAdapter verbose: server-b9e03c73-1a26-2c99-0894-c73344e94a9f: 2021-10-08T13:01:14.952Z [INFO] server-b9e03c73-1a26-2c99-0894-c73344e94a9f: entering leader state: leader="Node at b9e03c73-1a26-2c99-0894-c73344e94a9f [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) testing.go:149: server-b9e03c73-1a26-2c99-0894-c73344e94a9f: 2021-10-08T13:01:14.952Z [INFO] server-b9e03c73-1a26-2c99-0894-c73344e94a9f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b9e03c73-1a26-2c99-0894-c73344e94a9f Address:b9e03c73-1a26-2c99-0894-c73344e94a9f}]" testing.go:149: server-b9e03c73-1a26-2c99-0894-c73344e94a9f: 2021-10-08T13:01:14.952Z [INFO] server-b9e03c73-1a26-2c99-0894-c73344e94a9f: entering leader state: leader="Node at b9e03c73-1a26-2c99-0894-c73344e94a9f [Leader]" === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:14.952Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484 Address:b8bd46c9-8e95-a641-b4ce-d2845fe0b484}]" testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:14.952Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering follower state: follower="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Follower]" leader= testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [WARN] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering candidate state: node="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Candidate]" term=2 testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: election won: tally=1 testLoggerAdapter verbose: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering leader state: leader="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Leader]" --- PASS: TestRaft_NotifyCh (0.12s) testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:14.952Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484 Address:b8bd46c9-8e95-a641-b4ce-d2845fe0b484}]" testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:14.952Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering follower state: follower="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Follower]" leader= testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [WARN] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering candidate state: node="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Candidate]" term=2 testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: election won: tally=1 testing.go:149: server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: 2021-10-08T13:01:15.029Z [INFO] server-b8bd46c9-8e95-a641-b4ce-d2845fe0b484: entering leader state: leader="Node at b8bd46c9-8e95-a641-b4ce-d2845fe0b484 [Leader]" === RUN TestRaft_Voting testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.075Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.075Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering follower state: follower="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Follower]" leader= testLoggerAdapter verbose: server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: 2021-10-08T13:01:15.075Z [INFO] server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testLoggerAdapter verbose: server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: 2021-10-08T13:01:15.075Z [INFO] server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: entering follower state: follower="Node at e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 [Follower]" leader= testLoggerAdapter verbose: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.075Z [INFO] server-58b71da6-f34e-c1c6-1998-b4000e494b93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.075Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.076Z [INFO] server-58b71da6-f34e-c1c6-1998-b4000e494b93: entering follower state: follower="Node at 58b71da6-f34e-c1c6-1998-b4000e494b93 [Follower]" leader= testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [WARN] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering candidate state: node="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Candidate]" term=2 testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: election won: tally=2 testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering leader state: leader="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Leader]" testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: added peer, starting replication: peer=server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: added peer, starting replication: peer=server-58b71da6-f34e-c1c6-1998-b4000e494b93 testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: pipelining replication: peer="{Voter server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8}" testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: pipelining replication: peer="{Voter server-58b71da6-f34e-c1c6-1998-b4000e494b93 58b71da6-f34e-c1c6-1998-b4000e494b93}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.237Z [INFO] stable state for Follower reached at 2021-10-08 13:01:15.132821445 +0000 UTC m=+15.147504168 (2 nodes), 56.768443ms from start of poll, 57.546993ms from cluster start. Timeout at 2021-10-08 13:01:15.237914133 +0000 UTC m=+15.252596852, 105.092684ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.238Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.343Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.238028825 +0000 UTC m=+15.252711543 (1 nodes), 546ns from start of poll, 162.754368ms from cluster start. Timeout at 2021-10-08 13:01:15.343146582 +0000 UTC m=+15.357829306, 105.117763ms after stability testLoggerAdapter verbose: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.343Z [WARN] server-58b71da6-f34e-c1c6-1998-b4000e494b93: rejecting vote request since we have a leader: from=e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 leader=8afd72c0-8b72-d3b9-4aa5-b8db518de197 testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.343Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: aborting pipeline replication: peer="{Voter server-58b71da6-f34e-c1c6-1998-b4000e494b93 58b71da6-f34e-c1c6-1998-b4000e494b93}" testLoggerAdapter verbose: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.343Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: aborting pipeline replication: peer="{Voter server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8}" --- PASS: TestRaft_Voting (0.27s) testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.075Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.075Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering follower state: follower="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Follower]" leader= testing.go:149: server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: 2021-10-08T13:01:15.075Z [INFO] server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testing.go:149: server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: 2021-10-08T13:01:15.075Z [INFO] server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8: entering follower state: follower="Node at e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 [Follower]" leader= testing.go:149: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.075Z [INFO] server-58b71da6-f34e-c1c6-1998-b4000e494b93: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8afd72c0-8b72-d3b9-4aa5-b8db518de197 Address:8afd72c0-8b72-d3b9-4aa5-b8db518de197} {Suffrage:Voter ID:server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 Address:e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8} {Suffrage:Voter ID:server-58b71da6-f34e-c1c6-1998-b4000e494b93 Address:58b71da6-f34e-c1c6-1998-b4000e494b93}]" testing.go:149: cluster: 2021-10-08T13:01:15.075Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.076Z [INFO] server-58b71da6-f34e-c1c6-1998-b4000e494b93: entering follower state: follower="Node at 58b71da6-f34e-c1c6-1998-b4000e494b93 [Follower]" leader= testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [WARN] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering candidate state: node="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Candidate]" term=2 testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: election won: tally=2 testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: entering leader state: leader="Node at 8afd72c0-8b72-d3b9-4aa5-b8db518de197 [Leader]" testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: added peer, starting replication: peer=server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: added peer, starting replication: peer=server-58b71da6-f34e-c1c6-1998-b4000e494b93 testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: pipelining replication: peer="{Voter server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8}" testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.132Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: pipelining replication: peer="{Voter server-58b71da6-f34e-c1c6-1998-b4000e494b93 58b71da6-f34e-c1c6-1998-b4000e494b93}" testing.go:149: cluster: 2021-10-08T13:01:15.237Z [INFO] stable state for Follower reached at 2021-10-08 13:01:15.132821445 +0000 UTC m=+15.147504168 (2 nodes), 56.768443ms from start of poll, 57.546993ms from cluster start. Timeout at 2021-10-08 13:01:15.237914133 +0000 UTC m=+15.252596852, 105.092684ms after stability testing.go:149: cluster: 2021-10-08T13:01:15.238Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:15.343Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.238028825 +0000 UTC m=+15.252711543 (1 nodes), 546ns from start of poll, 162.754368ms from cluster start. Timeout at 2021-10-08 13:01:15.343146582 +0000 UTC m=+15.357829306, 105.117763ms after stability testing.go:149: server-58b71da6-f34e-c1c6-1998-b4000e494b93: 2021-10-08T13:01:15.343Z [WARN] server-58b71da6-f34e-c1c6-1998-b4000e494b93: rejecting vote request since we have a leader: from=e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 leader=8afd72c0-8b72-d3b9-4aa5-b8db518de197 testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.343Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: aborting pipeline replication: peer="{Voter server-58b71da6-f34e-c1c6-1998-b4000e494b93 58b71da6-f34e-c1c6-1998-b4000e494b93}" testing.go:149: server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: 2021-10-08T13:01:15.343Z [INFO] server-8afd72c0-8b72-d3b9-4aa5-b8db518de197: aborting pipeline replication: peer="{Voter server-e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8 e5ae3ebf-8a72-2ba5-4c33-4900137bf1d8}" === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.344Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testLoggerAdapter verbose: server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: 2021-10-08T13:01:15.345Z [INFO] server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.345Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering follower state: follower="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Follower]" leader= testLoggerAdapter verbose: server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: 2021-10-08T13:01:15.345Z [INFO] server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testLoggerAdapter verbose: server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: 2021-10-08T13:01:15.345Z [INFO] server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: entering follower state: follower="Node at ea0a83f5-2d5f-c0c8-4d65-706f580ca23f [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.345Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: 2021-10-08T13:01:15.345Z [INFO] server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: entering follower state: follower="Node at 29fb7b40-c8a4-df55-0a94-dd0434ea6dec [Follower]" leader= testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [WARN] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering candidate state: node="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Candidate]" term=2 testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: election won: tally=2 testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering leader state: leader="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Leader]" testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: added peer, starting replication: peer=server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: added peer, starting replication: peer=server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: pipelining replication: peer="{Voter server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f ea0a83f5-2d5f-c0c8-4d65-706f580ca23f}" testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: pipelining replication: peer="{Voter server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec 29fb7b40-c8a4-df55-0a94-dd0434ea6dec}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.520Z [INFO] stable state for Follower reached at 2021-10-08 13:01:15.415647139 +0000 UTC m=+15.430329909 (2 nodes), 70.413808ms from start of poll, 70.856635ms from cluster start. Timeout at 2021-10-08 13:01:15.520750519 +0000 UTC m=+15.535433243, 105.103334ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.520Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.626Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.520920535 +0000 UTC m=+15.535603254 (1 nodes), 646ns from start of poll, 176.12998ms from cluster start. Timeout at 2021-10-08 13:01:15.626011354 +0000 UTC m=+15.640694075, 105.090821ms after stability testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.626Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: aborting pipeline replication: peer="{Voter server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec 29fb7b40-c8a4-df55-0a94-dd0434ea6dec}" testLoggerAdapter verbose: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.626Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: aborting pipeline replication: peer="{Voter server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f ea0a83f5-2d5f-c0c8-4d65-706f580ca23f}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.28s) testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.344Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testing.go:149: server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: 2021-10-08T13:01:15.345Z [INFO] server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.345Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering follower state: follower="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Follower]" leader= testing.go:149: server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: 2021-10-08T13:01:15.345Z [INFO] server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2411c971-d247-9e66-d06f-f7aeeeff11e5 Address:2411c971-d247-9e66-d06f-f7aeeeff11e5} {Suffrage:Voter ID:server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f Address:ea0a83f5-2d5f-c0c8-4d65-706f580ca23f} {Suffrage:Voter ID:server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec Address:29fb7b40-c8a4-df55-0a94-dd0434ea6dec}]" testing.go:149: server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: 2021-10-08T13:01:15.345Z [INFO] server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f: entering follower state: follower="Node at ea0a83f5-2d5f-c0c8-4d65-706f580ca23f [Follower]" leader= testing.go:149: cluster: 2021-10-08T13:01:15.345Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: 2021-10-08T13:01:15.345Z [INFO] server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec: entering follower state: follower="Node at 29fb7b40-c8a4-df55-0a94-dd0434ea6dec [Follower]" leader= testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [WARN] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering candidate state: node="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Candidate]" term=2 testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: election won: tally=2 testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: entering leader state: leader="Node at 2411c971-d247-9e66-d06f-f7aeeeff11e5 [Leader]" testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: added peer, starting replication: peer=server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: added peer, starting replication: peer=server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: pipelining replication: peer="{Voter server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f ea0a83f5-2d5f-c0c8-4d65-706f580ca23f}" testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.415Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: pipelining replication: peer="{Voter server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec 29fb7b40-c8a4-df55-0a94-dd0434ea6dec}" testing.go:149: cluster: 2021-10-08T13:01:15.520Z [INFO] stable state for Follower reached at 2021-10-08 13:01:15.415647139 +0000 UTC m=+15.430329909 (2 nodes), 70.413808ms from start of poll, 70.856635ms from cluster start. Timeout at 2021-10-08 13:01:15.520750519 +0000 UTC m=+15.535433243, 105.103334ms after stability testing.go:149: cluster: 2021-10-08T13:01:15.520Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:15.626Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.520920535 +0000 UTC m=+15.535603254 (1 nodes), 646ns from start of poll, 176.12998ms from cluster start. Timeout at 2021-10-08 13:01:15.626011354 +0000 UTC m=+15.640694075, 105.090821ms after stability testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.626Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: aborting pipeline replication: peer="{Voter server-29fb7b40-c8a4-df55-0a94-dd0434ea6dec 29fb7b40-c8a4-df55-0a94-dd0434ea6dec}" testing.go:149: server-2411c971-d247-9e66-d06f-f7aeeeff11e5: 2021-10-08T13:01:15.626Z [INFO] server-2411c971-d247-9e66-d06f-f7aeeeff11e5: aborting pipeline replication: peer="{Voter server-ea0a83f5-2d5f-c0c8-4d65-706f580ca23f ea0a83f5-2d5f-c0c8-4d65-706f580ca23f}" === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.627Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testLoggerAdapter verbose: 75cb02e1-977a-9890-0fbf-0331b95ead80: 2021-10-08T13:01:15.627Z [INFO] 75cb02e1-977a-9890-0fbf-0331b95ead80: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.627Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering follower state: follower="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Follower]" leader= testLoggerAdapter verbose: 75cb02e1-977a-9890-0fbf-0331b95ead80: 2021-10-08T13:01:15.627Z [INFO] 75cb02e1-977a-9890-0fbf-0331b95ead80: entering follower state: follower="Node at 75cb02e1-977a-9890-0fbf-0331b95ead80 [Follower]" leader= testLoggerAdapter verbose: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.628Z [INFO] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.628Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.628Z [INFO] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: entering follower state: follower="Node at fcf81fef-f445-6ec4-9bbd-9fb5cd472435 [Follower]" leader= testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [WARN] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering candidate state: node="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Candidate]" term=2 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: election won: tally=2 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering leader state: leader="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Leader]" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: added peer, starting replication: peer=75cb02e1-977a-9890-0fbf-0331b95ead80 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: pipelining replication: peer="{Voter 75cb02e1-977a-9890-0fbf-0331b95ead80 75cb02e1-977a-9890-0fbf-0331b95ead80}" testLoggerAdapter verbose: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.695Z [WARN] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.784Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.679816477 +0000 UTC m=+15.694499221 (1 nodes), 51.619614ms from start of poll, 52.24251ms from cluster start. Timeout at 2021-10-08 13:01:15.7849262 +0000 UTC m=+15.799608944, 105.109723ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.785Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.890Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.785095831 +0000 UTC m=+15.799778562 (1 nodes), 1.117µs from start of poll, 157.521851ms from cluster start. Timeout at 2021-10-08 13:01:15.890211754 +0000 UTC m=+15.904894478, 105.115916ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.890Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:15.995Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.890350816 +0000 UTC m=+15.905033547 (1 nodes), 1µs from start of poll, 262.776836ms from cluster start. Timeout at 2021-10-08 13:01:15.995443698 +0000 UTC m=+16.010126418, 105.092871ms after stability testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: updating configuration: command=AddStaging server-id=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 server-addr=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80} {Suffrage:Voter ID:fcf81fef-f445-6ec4-9bbd-9fb5cd472435 Address:fcf81fef-f445-6ec4-9bbd-9fb5cd472435}]" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: added peer, starting replication: peer=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 testLoggerAdapter verbose: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.995Z [WARN] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [WARN] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: appendEntries rejected, sending older logs: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" next=1 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: pipelining replication: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.000Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.106Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.000942637 +0000 UTC m=+16.015625368 (1 nodes), 525ns from start of poll, 373.368657ms from cluster start. Timeout at 2021-10-08 13:01:16.106050457 +0000 UTC m=+16.120733177, 105.107809ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.106Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.211Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.106675627 +0000 UTC m=+16.121358364 (1 nodes), 1.27µs from start of poll, 479.101653ms from cluster start. Timeout at 2021-10-08 13:01:16.211766088 +0000 UTC m=+16.226448840, 105.090476ms after stability testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.211Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: updating configuration: command=RemoveServer server-id=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 server-addr= servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.211Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: removed peer, stopping replication: peer=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 last-index=4 testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.212Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: aborting pipeline replication: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" testLoggerAdapter verbose: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.212Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: aborting pipeline replication: peer="{Voter 75cb02e1-977a-9890-0fbf-0331b95ead80 75cb02e1-977a-9890-0fbf-0331b95ead80}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.59s) testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.627Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testing.go:149: 75cb02e1-977a-9890-0fbf-0331b95ead80: 2021-10-08T13:01:15.627Z [INFO] 75cb02e1-977a-9890-0fbf-0331b95ead80: initial configuration: index=1 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.627Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering follower state: follower="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Follower]" leader= testing.go:149: 75cb02e1-977a-9890-0fbf-0331b95ead80: 2021-10-08T13:01:15.627Z [INFO] 75cb02e1-977a-9890-0fbf-0331b95ead80: entering follower state: follower="Node at 75cb02e1-977a-9890-0fbf-0331b95ead80 [Follower]" leader= testing.go:149: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.628Z [INFO] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-08T13:01:15.628Z [INFO] starting stability test: raft-state=Leader testing.go:149: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.628Z [INFO] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: entering follower state: follower="Node at fcf81fef-f445-6ec4-9bbd-9fb5cd472435 [Follower]" leader= testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [WARN] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: heartbeat timeout reached, starting election: last-leader= testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering candidate state: node="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Candidate]" term=2 testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: election won: tally=2 testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: entering leader state: leader="Node at 3b98fa01-b1c0-4715-2dba-5ead48e9c823 [Leader]" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: added peer, starting replication: peer=75cb02e1-977a-9890-0fbf-0331b95ead80 testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.679Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: pipelining replication: peer="{Voter 75cb02e1-977a-9890-0fbf-0331b95ead80 75cb02e1-977a-9890-0fbf-0331b95ead80}" testing.go:149: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.695Z [WARN] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: no known peers, aborting election testing.go:149: cluster: 2021-10-08T13:01:15.784Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.679816477 +0000 UTC m=+15.694499221 (1 nodes), 51.619614ms from start of poll, 52.24251ms from cluster start. Timeout at 2021-10-08 13:01:15.7849262 +0000 UTC m=+15.799608944, 105.109723ms after stability testing.go:149: cluster: 2021-10-08T13:01:15.785Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:15.890Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.785095831 +0000 UTC m=+15.799778562 (1 nodes), 1.117µs from start of poll, 157.521851ms from cluster start. Timeout at 2021-10-08 13:01:15.890211754 +0000 UTC m=+15.904894478, 105.115916ms after stability testing.go:149: cluster: 2021-10-08T13:01:15.890Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:15.995Z [INFO] stable state for Leader reached at 2021-10-08 13:01:15.890350816 +0000 UTC m=+15.905033547 (1 nodes), 1µs from start of poll, 262.776836ms from cluster start. Timeout at 2021-10-08 13:01:15.995443698 +0000 UTC m=+16.010126418, 105.092871ms after stability testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: updating configuration: command=AddStaging server-id=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 server-addr=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80} {Suffrage:Voter ID:fcf81fef-f445-6ec4-9bbd-9fb5cd472435 Address:fcf81fef-f445-6ec4-9bbd-9fb5cd472435}]" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: added peer, starting replication: peer=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 testing.go:149: fcf81fef-f445-6ec4-9bbd-9fb5cd472435: 2021-10-08T13:01:15.995Z [WARN] fcf81fef-f445-6ec4-9bbd-9fb5cd472435: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [WARN] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: appendEntries rejected, sending older logs: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" next=1 testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:15.995Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: pipelining replication: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" testing.go:149: cluster: 2021-10-08T13:01:16.000Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:16.106Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.000942637 +0000 UTC m=+16.015625368 (1 nodes), 525ns from start of poll, 373.368657ms from cluster start. Timeout at 2021-10-08 13:01:16.106050457 +0000 UTC m=+16.120733177, 105.107809ms after stability testing.go:149: cluster: 2021-10-08T13:01:16.106Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:16.211Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.106675627 +0000 UTC m=+16.121358364 (1 nodes), 1.27µs from start of poll, 479.101653ms from cluster start. Timeout at 2021-10-08 13:01:16.211766088 +0000 UTC m=+16.226448840, 105.090476ms after stability testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.211Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: updating configuration: command=RemoveServer server-id=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 server-addr= servers="[{Suffrage:Voter ID:3b98fa01-b1c0-4715-2dba-5ead48e9c823 Address:3b98fa01-b1c0-4715-2dba-5ead48e9c823} {Suffrage:Voter ID:75cb02e1-977a-9890-0fbf-0331b95ead80 Address:75cb02e1-977a-9890-0fbf-0331b95ead80}]" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.211Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: removed peer, stopping replication: peer=fcf81fef-f445-6ec4-9bbd-9fb5cd472435 last-index=4 testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.212Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: aborting pipeline replication: peer="{Voter fcf81fef-f445-6ec4-9bbd-9fb5cd472435 fcf81fef-f445-6ec4-9bbd-9fb5cd472435}" testing.go:149: 3b98fa01-b1c0-4715-2dba-5ead48e9c823: 2021-10-08T13:01:16.212Z [INFO] 3b98fa01-b1c0-4715-2dba-5ead48e9c823: aborting pipeline replication: peer="{Voter 75cb02e1-977a-9890-0fbf-0331b95ead80 75cb02e1-977a-9890-0fbf-0331b95ead80}" === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: b977faba-421d-b467-1c52-33e56c227e44: 2021-10-08T13:01:16.213Z [INFO] b977faba-421d-b467-1c52-33e56c227e44: initial configuration: index=1 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0}]" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.213Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.213Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: b977faba-421d-b467-1c52-33e56c227e44: 2021-10-08T13:01:16.213Z [INFO] b977faba-421d-b467-1c52-33e56c227e44: entering follower state: follower="Node at b977faba-421d-b467-1c52-33e56c227e44 [Follower]" leader= testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.213Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering follower state: follower="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Follower]" leader= testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [WARN] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering candidate state: node="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Candidate]" term=2 testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: election won: tally=2 testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering leader state: leader="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Leader]" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: added peer, starting replication: peer=b977faba-421d-b467-1c52-33e56c227e44 testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: pipelining replication: peer="{Voter b977faba-421d-b467-1c52-33e56c227e44 b977faba-421d-b467-1c52-33e56c227e44}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.390Z [INFO] stable state for Follower reached at 2021-10-08 13:01:16.285729928 +0000 UTC m=+16.300412665 (1 nodes), 72.227407ms from start of poll, 72.49018ms from cluster start. Timeout at 2021-10-08 13:01:16.390805653 +0000 UTC m=+16.405488383, 105.075718ms after stability testLoggerAdapter verbose: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.391Z [INFO] server-5523125e-0868-3d68-c860-d859f1536973: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.391Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.391Z [INFO] server-5523125e-0868-3d68-c860-d859f1536973: entering follower state: follower="Node at 5523125e-0868-3d68-c860-d859f1536973 [Follower]" leader= testLoggerAdapter verbose: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.473Z [WARN] server-5523125e-0868-3d68-c860-d859f1536973: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.496Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.39146826 +0000 UTC m=+16.406150979 (1 nodes), 338ns from start of poll, 178.228494ms from cluster start. Timeout at 2021-10-08 13:01:16.496573346 +0000 UTC m=+16.511256071, 105.105092ms after stability testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: updating configuration: command=AddStaging server-id=server-5523125e-0868-3d68-c860-d859f1536973 server-addr=5523125e-0868-3d68-c860-d859f1536973 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0} {Suffrage:Voter ID:server-5523125e-0868-3d68-c860-d859f1536973 Address:5523125e-0868-3d68-c860-d859f1536973}]" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: added peer, starting replication: peer=server-5523125e-0868-3d68-c860-d859f1536973 testLoggerAdapter verbose: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.496Z [WARN] server-5523125e-0868-3d68-c860-d859f1536973: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [WARN] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: appendEntries rejected, sending older logs: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" next=1 testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.497Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: pipelining replication: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.502Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.607Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.502209252 +0000 UTC m=+16.516891979 (1 nodes), 732ns from start of poll, 288.969494ms from cluster start. Timeout at 2021-10-08 13:01:16.607258078 +0000 UTC m=+16.621940810, 105.048831ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.607Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.712Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.607417475 +0000 UTC m=+16.622100203 (1 nodes), 843ns from start of poll, 394.177718ms from cluster start. Timeout at 2021-10-08 13:01:16.712521834 +0000 UTC m=+16.727204558, 105.104355ms after stability testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: updating configuration: command=RemoveServer server-id=b977faba-421d-b467-1c52-33e56c227e44 server-addr= servers="[{Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0} {Suffrage:Voter ID:server-5523125e-0868-3d68-c860-d859f1536973 Address:5523125e-0868-3d68-c860-d859f1536973}]" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: removed peer, stopping replication: peer=b977faba-421d-b467-1c52-33e56c227e44 last-index=4 testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: aborting pipeline replication: peer="{Voter b977faba-421d-b467-1c52-33e56c227e44 b977faba-421d-b467-1c52-33e56c227e44}" testLoggerAdapter verbose: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: aborting pipeline replication: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.50s) testing.go:149: b977faba-421d-b467-1c52-33e56c227e44: 2021-10-08T13:01:16.213Z [INFO] b977faba-421d-b467-1c52-33e56c227e44: initial configuration: index=1 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0}]" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.213Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0}]" testing.go:149: cluster: 2021-10-08T13:01:16.213Z [INFO] starting stability test: raft-state=Follower testing.go:149: b977faba-421d-b467-1c52-33e56c227e44: 2021-10-08T13:01:16.213Z [INFO] b977faba-421d-b467-1c52-33e56c227e44: entering follower state: follower="Node at b977faba-421d-b467-1c52-33e56c227e44 [Follower]" leader= testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.213Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering follower state: follower="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Follower]" leader= testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [WARN] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: heartbeat timeout reached, starting election: last-leader= testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering candidate state: node="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Candidate]" term=2 testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: election won: tally=2 testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: entering leader state: leader="Node at 9eeaceb4-d61f-e53e-cfb0-710b225874b0 [Leader]" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: added peer, starting replication: peer=b977faba-421d-b467-1c52-33e56c227e44 testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.285Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: pipelining replication: peer="{Voter b977faba-421d-b467-1c52-33e56c227e44 b977faba-421d-b467-1c52-33e56c227e44}" testing.go:149: cluster: 2021-10-08T13:01:16.390Z [INFO] stable state for Follower reached at 2021-10-08 13:01:16.285729928 +0000 UTC m=+16.300412665 (1 nodes), 72.227407ms from start of poll, 72.49018ms from cluster start. Timeout at 2021-10-08 13:01:16.390805653 +0000 UTC m=+16.405488383, 105.075718ms after stability testing.go:149: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.391Z [INFO] server-5523125e-0868-3d68-c860-d859f1536973: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-08T13:01:16.391Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.391Z [INFO] server-5523125e-0868-3d68-c860-d859f1536973: entering follower state: follower="Node at 5523125e-0868-3d68-c860-d859f1536973 [Follower]" leader= testing.go:149: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.473Z [WARN] server-5523125e-0868-3d68-c860-d859f1536973: no known peers, aborting election testing.go:149: cluster: 2021-10-08T13:01:16.496Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.39146826 +0000 UTC m=+16.406150979 (1 nodes), 338ns from start of poll, 178.228494ms from cluster start. Timeout at 2021-10-08 13:01:16.496573346 +0000 UTC m=+16.511256071, 105.105092ms after stability testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: updating configuration: command=AddStaging server-id=server-5523125e-0868-3d68-c860-d859f1536973 server-addr=5523125e-0868-3d68-c860-d859f1536973 servers="[{Suffrage:Voter ID:b977faba-421d-b467-1c52-33e56c227e44 Address:b977faba-421d-b467-1c52-33e56c227e44} {Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0} {Suffrage:Voter ID:server-5523125e-0868-3d68-c860-d859f1536973 Address:5523125e-0868-3d68-c860-d859f1536973}]" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: added peer, starting replication: peer=server-5523125e-0868-3d68-c860-d859f1536973 testing.go:149: server-5523125e-0868-3d68-c860-d859f1536973: 2021-10-08T13:01:16.496Z [WARN] server-5523125e-0868-3d68-c860-d859f1536973: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.496Z [WARN] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: appendEntries rejected, sending older logs: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" next=1 testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.497Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: pipelining replication: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" testing.go:149: cluster: 2021-10-08T13:01:16.502Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:16.607Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.502209252 +0000 UTC m=+16.516891979 (1 nodes), 732ns from start of poll, 288.969494ms from cluster start. Timeout at 2021-10-08 13:01:16.607258078 +0000 UTC m=+16.621940810, 105.048831ms after stability testing.go:149: cluster: 2021-10-08T13:01:16.607Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:16.712Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.607417475 +0000 UTC m=+16.622100203 (1 nodes), 843ns from start of poll, 394.177718ms from cluster start. Timeout at 2021-10-08 13:01:16.712521834 +0000 UTC m=+16.727204558, 105.104355ms after stability testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: updating configuration: command=RemoveServer server-id=b977faba-421d-b467-1c52-33e56c227e44 server-addr= servers="[{Suffrage:Voter ID:9eeaceb4-d61f-e53e-cfb0-710b225874b0 Address:9eeaceb4-d61f-e53e-cfb0-710b225874b0} {Suffrage:Voter ID:server-5523125e-0868-3d68-c860-d859f1536973 Address:5523125e-0868-3d68-c860-d859f1536973}]" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: removed peer, stopping replication: peer=b977faba-421d-b467-1c52-33e56c227e44 last-index=4 testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: aborting pipeline replication: peer="{Voter b977faba-421d-b467-1c52-33e56c227e44 b977faba-421d-b467-1c52-33e56c227e44}" testing.go:149: 9eeaceb4-d61f-e53e-cfb0-710b225874b0: 2021-10-08T13:01:16.712Z [INFO] 9eeaceb4-d61f-e53e-cfb0-710b225874b0: aborting pipeline replication: peer="{Voter server-5523125e-0868-3d68-c860-d859f1536973 5523125e-0868-3d68-c860-d859f1536973}" === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.714Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.714Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.714Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering follower state: follower="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Follower]" leader= testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.714Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering follower state: follower="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Follower]" leader= testLoggerAdapter verbose: server-db6d4e11-94a9-aedd-87bc-e525764189da: 2021-10-08T13:01:16.714Z [INFO] server-db6d4e11-94a9-aedd-87bc-e525764189da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.714Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-db6d4e11-94a9-aedd-87bc-e525764189da: 2021-10-08T13:01:16.714Z [INFO] server-db6d4e11-94a9-aedd-87bc-e525764189da: entering follower state: follower="Node at db6d4e11-94a9-aedd-87bc-e525764189da [Follower]" leader= testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [WARN] server-a4781ae5-7635-a953-160d-2f93a89ea27b: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering candidate state: node="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Candidate]" term=2 testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: election won: tally=2 testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering leader state: leader="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Leader]" testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: added peer, starting replication: peer=server-4e02a4ff-e383-1738-af04-9122cb270d6d testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: added peer, starting replication: peer=server-db6d4e11-94a9-aedd-87bc-e525764189da testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: pipelining replication: peer="{Voter server-4e02a4ff-e383-1738-af04-9122cb270d6d 4e02a4ff-e383-1738-af04-9122cb270d6d}" testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: pipelining replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.877Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.772481751 +0000 UTC m=+16.787164494 (1 nodes), 57.792702ms from start of poll, 58.235402ms from cluster start. Timeout at 2021-10-08 13:01:16.877494091 +0000 UTC m=+16.892176832, 105.012338ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.877Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.982Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.877634911 +0000 UTC m=+16.892317639 (1 nodes), 810ns from start of poll, 163.388547ms from cluster start. Timeout at 2021-10-08 13:01:16.982711063 +0000 UTC m=+16.997393790, 105.076151ms after stability testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.982Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering candidate state: node="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:16.982Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.982Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: election won: tally=2 testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering leader state: leader="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Leader]" testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering follower state: follower="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Follower]" leader= testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: added peer, starting replication: peer=server-a4781ae5-7635-a953-160d-2f93a89ea27b testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: aborting pipeline replication: peer="{Voter server-4e02a4ff-e383-1738-af04-9122cb270d6d 4e02a4ff-e383-1738-af04-9122cb270d6d}" testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: added peer, starting replication: peer=server-db6d4e11-94a9-aedd-87bc-e525764189da testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: pipelining replication: peer="{Voter server-a4781ae5-7635-a953-160d-2f93a89ea27b a4781ae5-7635-a953-160d-2f93a89ea27b}" testLoggerAdapter verbose: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: aborting pipeline replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: pipelining replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.088Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.983112673 +0000 UTC m=+16.997795409 (1 nodes), 60.116µs from start of poll, 268.866317ms from cluster start. Timeout at 2021-10-08 13:01:17.088194975 +0000 UTC m=+17.102877712, 105.082303ms after stability testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:17.088Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: aborting pipeline replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testLoggerAdapter verbose: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:17.088Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: aborting pipeline replication: peer="{Voter server-a4781ae5-7635-a953-160d-2f93a89ea27b a4781ae5-7635-a953-160d-2f93a89ea27b}" --- PASS: TestRaft_LeadershipTransfer (0.38s) testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.714Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.714Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.714Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering follower state: follower="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Follower]" leader= testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.714Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering follower state: follower="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Follower]" leader= testing.go:149: server-db6d4e11-94a9-aedd-87bc-e525764189da: 2021-10-08T13:01:16.714Z [INFO] server-db6d4e11-94a9-aedd-87bc-e525764189da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-4e02a4ff-e383-1738-af04-9122cb270d6d Address:4e02a4ff-e383-1738-af04-9122cb270d6d} {Suffrage:Voter ID:server-a4781ae5-7635-a953-160d-2f93a89ea27b Address:a4781ae5-7635-a953-160d-2f93a89ea27b} {Suffrage:Voter ID:server-db6d4e11-94a9-aedd-87bc-e525764189da Address:db6d4e11-94a9-aedd-87bc-e525764189da}]" testing.go:149: cluster: 2021-10-08T13:01:16.714Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-db6d4e11-94a9-aedd-87bc-e525764189da: 2021-10-08T13:01:16.714Z [INFO] server-db6d4e11-94a9-aedd-87bc-e525764189da: entering follower state: follower="Node at db6d4e11-94a9-aedd-87bc-e525764189da [Follower]" leader= testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [WARN] server-a4781ae5-7635-a953-160d-2f93a89ea27b: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering candidate state: node="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Candidate]" term=2 testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: election won: tally=2 testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering leader state: leader="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Leader]" testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: added peer, starting replication: peer=server-4e02a4ff-e383-1738-af04-9122cb270d6d testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: added peer, starting replication: peer=server-db6d4e11-94a9-aedd-87bc-e525764189da testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: pipelining replication: peer="{Voter server-4e02a4ff-e383-1738-af04-9122cb270d6d 4e02a4ff-e383-1738-af04-9122cb270d6d}" testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.772Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: pipelining replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testing.go:149: cluster: 2021-10-08T13:01:16.877Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.772481751 +0000 UTC m=+16.787164494 (1 nodes), 57.792702ms from start of poll, 58.235402ms from cluster start. Timeout at 2021-10-08 13:01:16.877494091 +0000 UTC m=+16.892176832, 105.012338ms after stability testing.go:149: cluster: 2021-10-08T13:01:16.877Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:16.982Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.877634911 +0000 UTC m=+16.892317639 (1 nodes), 810ns from start of poll, 163.388547ms from cluster start. Timeout at 2021-10-08 13:01:16.982711063 +0000 UTC m=+16.997393790, 105.076151ms after stability testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.982Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering candidate state: node="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Candidate]" term=3 testing.go:149: cluster: 2021-10-08T13:01:16.982Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.982Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: election won: tally=2 testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: entering leader state: leader="Node at 4e02a4ff-e383-1738-af04-9122cb270d6d [Leader]" testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: entering follower state: follower="Node at a4781ae5-7635-a953-160d-2f93a89ea27b [Follower]" leader= testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: added peer, starting replication: peer=server-a4781ae5-7635-a953-160d-2f93a89ea27b testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: aborting pipeline replication: peer="{Voter server-4e02a4ff-e383-1738-af04-9122cb270d6d 4e02a4ff-e383-1738-af04-9122cb270d6d}" testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: added peer, starting replication: peer=server-db6d4e11-94a9-aedd-87bc-e525764189da testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: pipelining replication: peer="{Voter server-a4781ae5-7635-a953-160d-2f93a89ea27b a4781ae5-7635-a953-160d-2f93a89ea27b}" testing.go:149: server-a4781ae5-7635-a953-160d-2f93a89ea27b: 2021-10-08T13:01:16.983Z [INFO] server-a4781ae5-7635-a953-160d-2f93a89ea27b: aborting pipeline replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:16.983Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: pipelining replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testing.go:149: cluster: 2021-10-08T13:01:17.088Z [INFO] stable state for Leader reached at 2021-10-08 13:01:16.983112673 +0000 UTC m=+16.997795409 (1 nodes), 60.116µs from start of poll, 268.866317ms from cluster start. Timeout at 2021-10-08 13:01:17.088194975 +0000 UTC m=+17.102877712, 105.082303ms after stability testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:17.088Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: aborting pipeline replication: peer="{Voter server-db6d4e11-94a9-aedd-87bc-e525764189da db6d4e11-94a9-aedd-87bc-e525764189da}" testing.go:149: server-4e02a4ff-e383-1738-af04-9122cb270d6d: 2021-10-08T13:01:17.088Z [INFO] server-4e02a4ff-e383-1738-af04-9122cb270d6d: aborting pipeline replication: peer="{Voter server-a4781ae5-7635-a953-160d-2f93a89ea27b a4781ae5-7635-a953-160d-2f93a89ea27b}" === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.373Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39c52243-d8dd-29f4-1aef-0ba22c5127c8 Address:39c52243-d8dd-29f4-1aef-0ba22c5127c8}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.373Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.373Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering follower state: follower="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Follower]" leader= testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [WARN] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering candidate state: node="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Candidate]" term=2 testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: election won: tally=1 testLoggerAdapter verbose: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering leader state: leader="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Leader]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.575Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.470476113 +0000 UTC m=+17.485158857 (1 nodes), 96.650843ms from start of poll, 97.004574ms from cluster start. Timeout at 2021-10-08 13:01:17.575557972 +0000 UTC m=+17.590240703, 105.081846ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.49s) testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.373Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-39c52243-d8dd-29f4-1aef-0ba22c5127c8 Address:39c52243-d8dd-29f4-1aef-0ba22c5127c8}]" testing.go:149: cluster: 2021-10-08T13:01:17.373Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.373Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering follower state: follower="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Follower]" leader= testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [WARN] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering candidate state: node="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Candidate]" term=2 testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: election won: tally=1 testing.go:149: server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: 2021-10-08T13:01:17.470Z [INFO] server-39c52243-d8dd-29f4-1aef-0ba22c5127c8: entering leader state: leader="Node at 39c52243-d8dd-29f4-1aef-0ba22c5127c8 [Leader]" testing.go:149: cluster: 2021-10-08T13:01:17.575Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.470476113 +0000 UTC m=+17.485158857 (1 nodes), 96.650843ms from start of poll, 97.004574ms from cluster start. Timeout at 2021-10-08 13:01:17.575557972 +0000 UTC m=+17.590240703, 105.081846ms after stability === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.579Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.579Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering follower state: follower="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Follower]" leader= testLoggerAdapter verbose: server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: 2021-10-08T13:01:17.579Z [INFO] server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: 2021-10-08T13:01:17.579Z [INFO] server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: entering follower state: follower="Node at c0c7b7fd-5bf2-bb17-bd36-900915dedc98 [Follower]" leader= testLoggerAdapter verbose: server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: 2021-10-08T13:01:17.579Z [INFO] server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: 2021-10-08T13:01:17.579Z [INFO] server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: entering follower state: follower="Node at 2f46acc9-3c39-1fc7-2a70-e14158da6f37 [Follower]" leader= testLoggerAdapter verbose: server-07691384-3576-3544-a700-853d783b45b8: 2021-10-08T13:01:17.579Z [INFO] server-07691384-3576-3544-a700-853d783b45b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-07691384-3576-3544-a700-853d783b45b8: 2021-10-08T13:01:17.579Z [INFO] server-07691384-3576-3544-a700-853d783b45b8: entering follower state: follower="Node at 07691384-3576-3544-a700-853d783b45b8 [Follower]" leader= testLoggerAdapter verbose: server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: 2021-10-08T13:01:17.580Z [INFO] server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: 2021-10-08T13:01:17.580Z [INFO] server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: entering follower state: follower="Node at a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 [Follower]" leader= testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.580Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.580Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering follower state: follower="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Follower]" leader= testLoggerAdapter verbose: server-1fd22199-aa01-8d21-50ab-f5075c88dae7: 2021-10-08T13:01:17.580Z [INFO] server-1fd22199-aa01-8d21-50ab-f5075c88dae7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.580Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1fd22199-aa01-8d21-50ab-f5075c88dae7: 2021-10-08T13:01:17.580Z [INFO] server-1fd22199-aa01-8d21-50ab-f5075c88dae7: entering follower state: follower="Node at 1fd22199-aa01-8d21-50ab-f5075c88dae7 [Follower]" leader= testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [WARN] server-3b954d64-20f3-23d7-2661-709c5abc3fed: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering candidate state: node="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Candidate]" term=2 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: election won: tally=4 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering leader state: leader="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Leader]" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-a20980ef-8f82-1f76-4b81-630fbdb04d63 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-07691384-3576-3544-a700-853d783b45b8 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-1fd22199-aa01-8d21-50ab-f5075c88dae7 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-a20980ef-8f82-1f76-4b81-630fbdb04d63 a20980ef-8f82-1f76-4b81-630fbdb04d63}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.737Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.632611097 +0000 UTC m=+17.647293838 (1 nodes), 52.009497ms from start of poll, 53.584158ms from cluster start. Timeout at 2021-10-08 13:01:17.737706598 +0000 UTC m=+17.752389316, 105.095478ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.737Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.843Z [INFO] stable state for Follower reached at 2021-10-08 13:01:17.737899959 +0000 UTC m=+17.752582681 (6 nodes), 1.405µs from start of poll, 158.873001ms from cluster start. Timeout at 2021-10-08 13:01:17.842968303 +0000 UTC m=+17.857651031, 105.06835ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.843Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.948Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.843072202 +0000 UTC m=+17.857754923 (1 nodes), 672ns from start of poll, 264.045243ms from cluster start. Timeout at 2021-10-08 13:01:17.948132944 +0000 UTC m=+17.962815668, 105.060745ms after stability testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering candidate state: node="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:17.948Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: election won: tally=4 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering follower state: follower="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Follower]" leader= testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering leader state: leader="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Leader]" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-07691384-3576-3544-a700-853d783b45b8 testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-a20980ef-8f82-1f76-4b81-630fbdb04d63 a20980ef-8f82-1f76-4b81-630fbdb04d63}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-3b954d64-20f3-23d7-2661-709c5abc3fed testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-1fd22199-aa01-8d21-50ab-f5075c88dae7 testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-3b954d64-20f3-23d7-2661-709c5abc3fed 3b954d64-20f3-23d7-2661-709c5abc3fed}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testLoggerAdapter verbose: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.053Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.94843032 +0000 UTC m=+17.963113037 (1 nodes), 37.777µs from start of poll, 369.403357ms from cluster start. Timeout at 2021-10-08 13:01:18.053556378 +0000 UTC m=+18.068239124, 105.126087ms after stability testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-3b954d64-20f3-23d7-2661-709c5abc3fed 3b954d64-20f3-23d7-2661-709c5abc3fed}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testLoggerAdapter verbose: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.579Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.579Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering follower state: follower="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Follower]" leader= testing.go:149: server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: 2021-10-08T13:01:17.579Z [INFO] server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: 2021-10-08T13:01:17.579Z [INFO] server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98: entering follower state: follower="Node at c0c7b7fd-5bf2-bb17-bd36-900915dedc98 [Follower]" leader= testing.go:149: server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: 2021-10-08T13:01:17.579Z [INFO] server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: 2021-10-08T13:01:17.579Z [INFO] server-2f46acc9-3c39-1fc7-2a70-e14158da6f37: entering follower state: follower="Node at 2f46acc9-3c39-1fc7-2a70-e14158da6f37 [Follower]" leader= testing.go:149: server-07691384-3576-3544-a700-853d783b45b8: 2021-10-08T13:01:17.579Z [INFO] server-07691384-3576-3544-a700-853d783b45b8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-07691384-3576-3544-a700-853d783b45b8: 2021-10-08T13:01:17.579Z [INFO] server-07691384-3576-3544-a700-853d783b45b8: entering follower state: follower="Node at 07691384-3576-3544-a700-853d783b45b8 [Follower]" leader= testing.go:149: server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: 2021-10-08T13:01:17.580Z [INFO] server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: 2021-10-08T13:01:17.580Z [INFO] server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7: entering follower state: follower="Node at a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 [Follower]" leader= testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.580Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.580Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering follower state: follower="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Follower]" leader= testing.go:149: server-1fd22199-aa01-8d21-50ab-f5075c88dae7: 2021-10-08T13:01:17.580Z [INFO] server-1fd22199-aa01-8d21-50ab-f5075c88dae7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a20980ef-8f82-1f76-4b81-630fbdb04d63 Address:a20980ef-8f82-1f76-4b81-630fbdb04d63} {Suffrage:Voter ID:server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 Address:c0c7b7fd-5bf2-bb17-bd36-900915dedc98} {Suffrage:Voter ID:server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 Address:2f46acc9-3c39-1fc7-2a70-e14158da6f37} {Suffrage:Voter ID:server-07691384-3576-3544-a700-853d783b45b8 Address:07691384-3576-3544-a700-853d783b45b8} {Suffrage:Voter ID:server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 Address:a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7} {Suffrage:Voter ID:server-3b954d64-20f3-23d7-2661-709c5abc3fed Address:3b954d64-20f3-23d7-2661-709c5abc3fed} {Suffrage:Voter ID:server-1fd22199-aa01-8d21-50ab-f5075c88dae7 Address:1fd22199-aa01-8d21-50ab-f5075c88dae7}]" testing.go:149: cluster: 2021-10-08T13:01:17.580Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-1fd22199-aa01-8d21-50ab-f5075c88dae7: 2021-10-08T13:01:17.580Z [INFO] server-1fd22199-aa01-8d21-50ab-f5075c88dae7: entering follower state: follower="Node at 1fd22199-aa01-8d21-50ab-f5075c88dae7 [Follower]" leader= testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [WARN] server-3b954d64-20f3-23d7-2661-709c5abc3fed: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering candidate state: node="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Candidate]" term=2 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: election won: tally=4 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering leader state: leader="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Leader]" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-a20980ef-8f82-1f76-4b81-630fbdb04d63 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-07691384-3576-3544-a700-853d783b45b8 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: added peer, starting replication: peer=server-1fd22199-aa01-8d21-50ab-f5075c88dae7 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.632Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: pipelining replication: peer="{Voter server-a20980ef-8f82-1f76-4b81-630fbdb04d63 a20980ef-8f82-1f76-4b81-630fbdb04d63}" testing.go:149: cluster: 2021-10-08T13:01:17.737Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.632611097 +0000 UTC m=+17.647293838 (1 nodes), 52.009497ms from start of poll, 53.584158ms from cluster start. Timeout at 2021-10-08 13:01:17.737706598 +0000 UTC m=+17.752389316, 105.095478ms after stability testing.go:149: cluster: 2021-10-08T13:01:17.737Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:17.843Z [INFO] stable state for Follower reached at 2021-10-08 13:01:17.737899959 +0000 UTC m=+17.752582681 (6 nodes), 1.405µs from start of poll, 158.873001ms from cluster start. Timeout at 2021-10-08 13:01:17.842968303 +0000 UTC m=+17.857651031, 105.06835ms after stability testing.go:149: cluster: 2021-10-08T13:01:17.843Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:17.948Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.843072202 +0000 UTC m=+17.857754923 (1 nodes), 672ns from start of poll, 264.045243ms from cluster start. Timeout at 2021-10-08 13:01:17.948132944 +0000 UTC m=+17.962815668, 105.060745ms after stability testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering candidate state: node="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Candidate]" term=3 testing.go:149: cluster: 2021-10-08T13:01:17.948Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: election won: tally=4 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: entering follower state: follower="Node at 3b954d64-20f3-23d7-2661-709c5abc3fed [Follower]" leader= testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: entering leader state: leader="Node at a20980ef-8f82-1f76-4b81-630fbdb04d63 [Leader]" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-07691384-3576-3544-a700-853d783b45b8 testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-a20980ef-8f82-1f76-4b81-630fbdb04d63 a20980ef-8f82-1f76-4b81-630fbdb04d63}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-3b954d64-20f3-23d7-2661-709c5abc3fed testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: added peer, starting replication: peer=server-1fd22199-aa01-8d21-50ab-f5075c88dae7 testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-3b954d64-20f3-23d7-2661-709c5abc3fed 3b954d64-20f3-23d7-2661-709c5abc3fed}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testing.go:149: server-3b954d64-20f3-23d7-2661-709c5abc3fed: 2021-10-08T13:01:17.948Z [INFO] server-3b954d64-20f3-23d7-2661-709c5abc3fed: aborting pipeline replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:17.948Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: pipelining replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testing.go:149: cluster: 2021-10-08T13:01:18.053Z [INFO] stable state for Leader reached at 2021-10-08 13:01:17.94843032 +0000 UTC m=+17.963113037 (1 nodes), 37.777µs from start of poll, 369.403357ms from cluster start. Timeout at 2021-10-08 13:01:18.053556378 +0000 UTC m=+18.068239124, 105.126087ms after stability testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7 a7e9d62c-6ae2-5c49-2a2a-edbf9b478fd7}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-3b954d64-20f3-23d7-2661-709c5abc3fed 3b954d64-20f3-23d7-2661-709c5abc3fed}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-2f46acc9-3c39-1fc7-2a70-e14158da6f37 2f46acc9-3c39-1fc7-2a70-e14158da6f37}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-1fd22199-aa01-8d21-50ab-f5075c88dae7 1fd22199-aa01-8d21-50ab-f5075c88dae7}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-07691384-3576-3544-a700-853d783b45b8 07691384-3576-3544-a700-853d783b45b8}" testing.go:149: server-a20980ef-8f82-1f76-4b81-630fbdb04d63: 2021-10-08T13:01:18.053Z [INFO] server-a20980ef-8f82-1f76-4b81-630fbdb04d63: aborting pipeline replication: peer="{Voter server-c0c7b7fd-5bf2-bb17-bd36-900915dedc98 c0c7b7fd-5bf2-bb17-bd36-900915dedc98}" === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-0490dab7-56b8-c70d-c723-6982f77f79d3: 2021-10-08T13:01:18.056Z [INFO] server-0490dab7-56b8-c70d-c723-6982f77f79d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.056Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testLoggerAdapter verbose: server-0490dab7-56b8-c70d-c723-6982f77f79d3: 2021-10-08T13:01:18.056Z [INFO] server-0490dab7-56b8-c70d-c723-6982f77f79d3: entering follower state: follower="Node at 0490dab7-56b8-c70d-c723-6982f77f79d3 [Follower]" leader= testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.056Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering follower state: follower="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Follower]" leader= testLoggerAdapter verbose: server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: 2021-10-08T13:01:18.056Z [INFO] server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.056Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: 2021-10-08T13:01:18.056Z [INFO] server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: entering follower state: follower="Node at 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 [Follower]" leader= testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [WARN] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering candidate state: node="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Candidate]" term=2 testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: election won: tally=2 testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering leader state: leader="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Leader]" testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: added peer, starting replication: peer=server-0490dab7-56b8-c70d-c723-6982f77f79d3 testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: added peer, starting replication: peer=server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: pipelining replication: peer="{Voter server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}" testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: pipelining replication: peer="{Voter server-0490dab7-56b8-c70d-c723-6982f77f79d3 0490dab7-56b8-c70d-c723-6982f77f79d3}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.223Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.118093276 +0000 UTC m=+18.132776025 (1 nodes), 61.226944ms from start of poll, 61.694812ms from cluster start. Timeout at 2021-10-08 13:01:18.223193765 +0000 UTC m=+18.237876486, 105.100461ms after stability testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.223Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: aborting pipeline replication: peer="{Voter server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}" testLoggerAdapter verbose: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.223Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: aborting pipeline replication: peer="{Voter server-0490dab7-56b8-c70d-c723-6982f77f79d3 0490dab7-56b8-c70d-c723-6982f77f79d3}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) testing.go:149: server-0490dab7-56b8-c70d-c723-6982f77f79d3: 2021-10-08T13:01:18.056Z [INFO] server-0490dab7-56b8-c70d-c723-6982f77f79d3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.056Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testing.go:149: server-0490dab7-56b8-c70d-c723-6982f77f79d3: 2021-10-08T13:01:18.056Z [INFO] server-0490dab7-56b8-c70d-c723-6982f77f79d3: entering follower state: follower="Node at 0490dab7-56b8-c70d-c723-6982f77f79d3 [Follower]" leader= testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.056Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering follower state: follower="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Follower]" leader= testing.go:149: server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: 2021-10-08T13:01:18.056Z [INFO] server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0490dab7-56b8-c70d-c723-6982f77f79d3 Address:0490dab7-56b8-c70d-c723-6982f77f79d3} {Suffrage:Voter ID:server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171 Address:0dd1ff21-0c43-cfd6-4243-87f9f0a49171} {Suffrage:Voter ID:server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 Address:1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}]" testing.go:149: cluster: 2021-10-08T13:01:18.056Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: 2021-10-08T13:01:18.056Z [INFO] server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844: entering follower state: follower="Node at 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 [Follower]" leader= testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [WARN] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering candidate state: node="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Candidate]" term=2 testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: election won: tally=2 testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.117Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: entering leader state: leader="Node at 0dd1ff21-0c43-cfd6-4243-87f9f0a49171 [Leader]" testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: added peer, starting replication: peer=server-0490dab7-56b8-c70d-c723-6982f77f79d3 testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: added peer, starting replication: peer=server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: pipelining replication: peer="{Voter server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}" testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.118Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: pipelining replication: peer="{Voter server-0490dab7-56b8-c70d-c723-6982f77f79d3 0490dab7-56b8-c70d-c723-6982f77f79d3}" testing.go:149: cluster: 2021-10-08T13:01:18.223Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.118093276 +0000 UTC m=+18.132776025 (1 nodes), 61.226944ms from start of poll, 61.694812ms from cluster start. Timeout at 2021-10-08 13:01:18.223193765 +0000 UTC m=+18.237876486, 105.100461ms after stability testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.223Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: aborting pipeline replication: peer="{Voter server-1ab3fba7-bb5a-8253-7265-f0bf0b5d9844 1ab3fba7-bb5a-8253-7265-f0bf0b5d9844}" testing.go:149: server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: 2021-10-08T13:01:18.223Z [INFO] server-0dd1ff21-0c43-cfd6-4243-87f9f0a49171: aborting pipeline replication: peer="{Voter server-0490dab7-56b8-c70d-c723-6982f77f79d3 0490dab7-56b8-c70d-c723-6982f77f79d3}" === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: 2021-10-08T13:01:18.225Z [INFO] server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testLoggerAdapter verbose: server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: 2021-10-08T13:01:18.225Z [INFO] server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: entering follower state: follower="Node at 0e43cdfc-ca2a-4691-1b28-11f623fc65a9 [Follower]" leader= testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.225Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.225Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering follower state: follower="Node at e486367a-1327-a273-0c37-99250605d431 [Follower]" leader= testLoggerAdapter verbose: server-6d524f33-9748-b32f-6116-bb08c3515adc: 2021-10-08T13:01:18.226Z [INFO] server-6d524f33-9748-b32f-6116-bb08c3515adc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.226Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-6d524f33-9748-b32f-6116-bb08c3515adc: 2021-10-08T13:01:18.226Z [INFO] server-6d524f33-9748-b32f-6116-bb08c3515adc: entering follower state: follower="Node at 6d524f33-9748-b32f-6116-bb08c3515adc [Follower]" leader= testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [WARN] server-e486367a-1327-a273-0c37-99250605d431: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering candidate state: node="Node at e486367a-1327-a273-0c37-99250605d431 [Candidate]" term=2 testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: election won: tally=2 testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering leader state: leader="Node at e486367a-1327-a273-0c37-99250605d431 [Leader]" testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: added peer, starting replication: peer=server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: added peer, starting replication: peer=server-6d524f33-9748-b32f-6116-bb08c3515adc testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: pipelining replication: peer="{Voter server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 0e43cdfc-ca2a-4691-1b28-11f623fc65a9}" testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: pipelining replication: peer="{Voter server-6d524f33-9748-b32f-6116-bb08c3515adc 6d524f33-9748-b32f-6116-bb08c3515adc}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.385Z [INFO] stable state for Follower reached at 2021-10-08 13:01:18.280311354 +0000 UTC m=+18.294994090 (2 nodes), 54.215331ms from start of poll, 54.861436ms from cluster start. Timeout at 2021-10-08 13:01:18.385390388 +0000 UTC m=+18.400073116, 105.079026ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.385Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.490Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.38555618 +0000 UTC m=+18.400238899 (1 nodes), 988ns from start of poll, 160.106245ms from cluster start. Timeout at 2021-10-08 13:01:18.490639965 +0000 UTC m=+18.505322702, 105.083803ms after stability testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.490Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: aborting pipeline replication: peer="{Voter server-6d524f33-9748-b32f-6116-bb08c3515adc 6d524f33-9748-b32f-6116-bb08c3515adc}" testLoggerAdapter verbose: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.490Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: aborting pipeline replication: peer="{Voter server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 0e43cdfc-ca2a-4691-1b28-11f623fc65a9}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.27s) testing.go:149: server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: 2021-10-08T13:01:18.225Z [INFO] server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testing.go:149: server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: 2021-10-08T13:01:18.225Z [INFO] server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9: entering follower state: follower="Node at 0e43cdfc-ca2a-4691-1b28-11f623fc65a9 [Follower]" leader= testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.225Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.225Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering follower state: follower="Node at e486367a-1327-a273-0c37-99250605d431 [Follower]" leader= testing.go:149: server-6d524f33-9748-b32f-6116-bb08c3515adc: 2021-10-08T13:01:18.226Z [INFO] server-6d524f33-9748-b32f-6116-bb08c3515adc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 Address:0e43cdfc-ca2a-4691-1b28-11f623fc65a9} {Suffrage:Voter ID:server-e486367a-1327-a273-0c37-99250605d431 Address:e486367a-1327-a273-0c37-99250605d431} {Suffrage:Voter ID:server-6d524f33-9748-b32f-6116-bb08c3515adc Address:6d524f33-9748-b32f-6116-bb08c3515adc}]" testing.go:149: cluster: 2021-10-08T13:01:18.226Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-6d524f33-9748-b32f-6116-bb08c3515adc: 2021-10-08T13:01:18.226Z [INFO] server-6d524f33-9748-b32f-6116-bb08c3515adc: entering follower state: follower="Node at 6d524f33-9748-b32f-6116-bb08c3515adc [Follower]" leader= testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [WARN] server-e486367a-1327-a273-0c37-99250605d431: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering candidate state: node="Node at e486367a-1327-a273-0c37-99250605d431 [Candidate]" term=2 testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: election won: tally=2 testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: entering leader state: leader="Node at e486367a-1327-a273-0c37-99250605d431 [Leader]" testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: added peer, starting replication: peer=server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: added peer, starting replication: peer=server-6d524f33-9748-b32f-6116-bb08c3515adc testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: pipelining replication: peer="{Voter server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 0e43cdfc-ca2a-4691-1b28-11f623fc65a9}" testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.280Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: pipelining replication: peer="{Voter server-6d524f33-9748-b32f-6116-bb08c3515adc 6d524f33-9748-b32f-6116-bb08c3515adc}" testing.go:149: cluster: 2021-10-08T13:01:18.385Z [INFO] stable state for Follower reached at 2021-10-08 13:01:18.280311354 +0000 UTC m=+18.294994090 (2 nodes), 54.215331ms from start of poll, 54.861436ms from cluster start. Timeout at 2021-10-08 13:01:18.385390388 +0000 UTC m=+18.400073116, 105.079026ms after stability testing.go:149: cluster: 2021-10-08T13:01:18.385Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:18.490Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.38555618 +0000 UTC m=+18.400238899 (1 nodes), 988ns from start of poll, 160.106245ms from cluster start. Timeout at 2021-10-08 13:01:18.490639965 +0000 UTC m=+18.505322702, 105.083803ms after stability testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.490Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: aborting pipeline replication: peer="{Voter server-6d524f33-9748-b32f-6116-bb08c3515adc 6d524f33-9748-b32f-6116-bb08c3515adc}" testing.go:149: server-e486367a-1327-a273-0c37-99250605d431: 2021-10-08T13:01:18.490Z [INFO] server-e486367a-1327-a273-0c37-99250605d431: aborting pipeline replication: peer="{Voter server-0e43cdfc-ca2a-4691-1b28-11f623fc65a9 0e43cdfc-ca2a-4691-1b28-11f623fc65a9}" === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.493Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.493Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering follower state: follower="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Follower]" leader= testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.493Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.493Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering follower state: follower="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Follower]" leader= testLoggerAdapter verbose: server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: 2021-10-08T13:01:18.493Z [INFO] server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.493Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: 2021-10-08T13:01:18.493Z [INFO] server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: entering follower state: follower="Node at 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d [Follower]" leader= testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.547Z [WARN] server-b01965aa-0fda-6612-167b-ae3ce14166b4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering candidate state: node="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Candidate]" term=2 testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: election won: tally=2 testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering leader state: leader="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Leader]" testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: added peer, starting replication: peer=server-e9f8817d-88f3-2dfe-e127-482db5dc667c testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: added peer, starting replication: peer=server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: pipelining replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: pipelining replication: peer="{Voter server-e9f8817d-88f3-2dfe-e127-482db5dc667c e9f8817d-88f3-2dfe-e127-482db5dc667c}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.653Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.548229531 +0000 UTC m=+18.562912251 (1 nodes), 54.647393ms from start of poll, 55.325077ms from cluster start. Timeout at 2021-10-08 13:01:18.653340576 +0000 UTC m=+18.668023316, 105.111065ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.653Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.758Z [INFO] stable state for Follower reached at 2021-10-08 13:01:18.653490408 +0000 UTC m=+18.668173195 (2 nodes), 1.1µs from start of poll, 160.586021ms from cluster start. Timeout at 2021-10-08 13:01:18.758625431 +0000 UTC m=+18.773308171, 105.134976ms after stability testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering candidate state: node="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.780Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: election won: tally=2 testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering follower state: follower="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Follower]" leader= testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering leader state: leader="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Leader]" testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: added peer, starting replication: peer=server-b01965aa-0fda-6612-167b-ae3ce14166b4 testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: added peer, starting replication: peer=server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: aborting pipeline replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: pipelining replication: peer="{Voter server-b01965aa-0fda-6612-167b-ae3ce14166b4 b01965aa-0fda-6612-167b-ae3ce14166b4}" testLoggerAdapter verbose: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: aborting pipeline replication: peer="{Voter server-e9f8817d-88f3-2dfe-e127-482db5dc667c e9f8817d-88f3-2dfe-e127-482db5dc667c}" testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: pipelining replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.885Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.780368395 +0000 UTC m=+18.795051120 (1 nodes), 122.183µs from start of poll, 287.463946ms from cluster start. Timeout at 2021-10-08 13:01:18.885445289 +0000 UTC m=+18.900128014, 105.076894ms after stability testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.885Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: aborting pipeline replication: peer="{Voter server-b01965aa-0fda-6612-167b-ae3ce14166b4 b01965aa-0fda-6612-167b-ae3ce14166b4}" testLoggerAdapter verbose: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.885Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: aborting pipeline replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.39s) testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.493Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.493Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering follower state: follower="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Follower]" leader= testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.493Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.493Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering follower state: follower="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Follower]" leader= testing.go:149: server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: 2021-10-08T13:01:18.493Z [INFO] server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b01965aa-0fda-6612-167b-ae3ce14166b4 Address:b01965aa-0fda-6612-167b-ae3ce14166b4} {Suffrage:Voter ID:server-e9f8817d-88f3-2dfe-e127-482db5dc667c Address:e9f8817d-88f3-2dfe-e127-482db5dc667c} {Suffrage:Voter ID:server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d Address:74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}]" testing.go:149: cluster: 2021-10-08T13:01:18.493Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: 2021-10-08T13:01:18.493Z [INFO] server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d: entering follower state: follower="Node at 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d [Follower]" leader= testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.547Z [WARN] server-b01965aa-0fda-6612-167b-ae3ce14166b4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering candidate state: node="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Candidate]" term=2 testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: election won: tally=2 testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering leader state: leader="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Leader]" testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: added peer, starting replication: peer=server-e9f8817d-88f3-2dfe-e127-482db5dc667c testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: added peer, starting replication: peer=server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: pipelining replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.548Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: pipelining replication: peer="{Voter server-e9f8817d-88f3-2dfe-e127-482db5dc667c e9f8817d-88f3-2dfe-e127-482db5dc667c}" testing.go:149: cluster: 2021-10-08T13:01:18.653Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.548229531 +0000 UTC m=+18.562912251 (1 nodes), 54.647393ms from start of poll, 55.325077ms from cluster start. Timeout at 2021-10-08 13:01:18.653340576 +0000 UTC m=+18.668023316, 105.111065ms after stability testing.go:149: cluster: 2021-10-08T13:01:18.653Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:18.758Z [INFO] stable state for Follower reached at 2021-10-08 13:01:18.653490408 +0000 UTC m=+18.668173195 (2 nodes), 1.1µs from start of poll, 160.586021ms from cluster start. Timeout at 2021-10-08 13:01:18.758625431 +0000 UTC m=+18.773308171, 105.134976ms after stability testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering candidate state: node="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Candidate]" term=3 testing.go:149: cluster: 2021-10-08T13:01:18.780Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: election won: tally=2 testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: entering follower state: follower="Node at b01965aa-0fda-6612-167b-ae3ce14166b4 [Follower]" leader= testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: entering leader state: leader="Node at e9f8817d-88f3-2dfe-e127-482db5dc667c [Leader]" testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: added peer, starting replication: peer=server-b01965aa-0fda-6612-167b-ae3ce14166b4 testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: added peer, starting replication: peer=server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: aborting pipeline replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: pipelining replication: peer="{Voter server-b01965aa-0fda-6612-167b-ae3ce14166b4 b01965aa-0fda-6612-167b-ae3ce14166b4}" testing.go:149: server-b01965aa-0fda-6612-167b-ae3ce14166b4: 2021-10-08T13:01:18.780Z [INFO] server-b01965aa-0fda-6612-167b-ae3ce14166b4: aborting pipeline replication: peer="{Voter server-e9f8817d-88f3-2dfe-e127-482db5dc667c e9f8817d-88f3-2dfe-e127-482db5dc667c}" testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.780Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: pipelining replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" testing.go:149: cluster: 2021-10-08T13:01:18.885Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.780368395 +0000 UTC m=+18.795051120 (1 nodes), 122.183µs from start of poll, 287.463946ms from cluster start. Timeout at 2021-10-08 13:01:18.885445289 +0000 UTC m=+18.900128014, 105.076894ms after stability testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.885Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: aborting pipeline replication: peer="{Voter server-b01965aa-0fda-6612-167b-ae3ce14166b4 b01965aa-0fda-6612-167b-ae3ce14166b4}" testing.go:149: server-e9f8817d-88f3-2dfe-e127-482db5dc667c: 2021-10-08T13:01:18.885Z [INFO] server-e9f8817d-88f3-2dfe-e127-482db5dc667c: aborting pipeline replication: peer="{Voter server-74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d 74f1f521-e717-d9d2-56ea-3bc0ea0f2e6d}" === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.887Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.887Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering follower state: follower="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Follower]" leader= testLoggerAdapter verbose: server-813c3757-1864-5f0c-0b36-edbd775f2cfd: 2021-10-08T13:01:18.887Z [INFO] server-813c3757-1864-5f0c-0b36-edbd775f2cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testLoggerAdapter verbose: server-813c3757-1864-5f0c-0b36-edbd775f2cfd: 2021-10-08T13:01:18.887Z [INFO] server-813c3757-1864-5f0c-0b36-edbd775f2cfd: entering follower state: follower="Node at 813c3757-1864-5f0c-0b36-edbd775f2cfd [Follower]" leader= testLoggerAdapter verbose: server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: 2021-10-08T13:01:18.887Z [INFO] server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:18.887Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: 2021-10-08T13:01:18.887Z [INFO] server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: entering follower state: follower="Node at 94a4bc19-2b78-2df4-9cfc-9854e4dbb934 [Follower]" leader= testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [WARN] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering candidate state: node="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Candidate]" term=2 testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: election won: tally=2 testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering leader state: leader="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Leader]" testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: added peer, starting replication: peer=server-813c3757-1864-5f0c-0b36-edbd775f2cfd testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: added peer, starting replication: peer=server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: pipelining replication: peer="{Voter server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 94a4bc19-2b78-2df4-9cfc-9854e4dbb934}" testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: pipelining replication: peer="{Voter server-813c3757-1864-5f0c-0b36-edbd775f2cfd 813c3757-1864-5f0c-0b36-edbd775f2cfd}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.072Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.96710849 +0000 UTC m=+18.981791242 (1 nodes), 79.571984ms from start of poll, 80.086809ms from cluster start. Timeout at 2021-10-08 13:01:19.072199941 +0000 UTC m=+19.086882683, 105.091441ms after stability testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: cannot transfer leadership to itself testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: aborting pipeline replication: peer="{Voter server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 94a4bc19-2b78-2df4-9cfc-9854e4dbb934}" testLoggerAdapter verbose: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: aborting pipeline replication: peer="{Voter server-813c3757-1864-5f0c-0b36-edbd775f2cfd 813c3757-1864-5f0c-0b36-edbd775f2cfd}" --- PASS: TestRaft_LeadershipTransferToItself (0.19s) testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.887Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.887Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering follower state: follower="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Follower]" leader= testing.go:149: server-813c3757-1864-5f0c-0b36-edbd775f2cfd: 2021-10-08T13:01:18.887Z [INFO] server-813c3757-1864-5f0c-0b36-edbd775f2cfd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testing.go:149: server-813c3757-1864-5f0c-0b36-edbd775f2cfd: 2021-10-08T13:01:18.887Z [INFO] server-813c3757-1864-5f0c-0b36-edbd775f2cfd: entering follower state: follower="Node at 813c3757-1864-5f0c-0b36-edbd775f2cfd [Follower]" leader= testing.go:149: server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: 2021-10-08T13:01:18.887Z [INFO] server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78 Address:a70513bc-bebd-07bc-2f5f-77d0d59c2e78} {Suffrage:Voter ID:server-813c3757-1864-5f0c-0b36-edbd775f2cfd Address:813c3757-1864-5f0c-0b36-edbd775f2cfd} {Suffrage:Voter ID:server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 Address:94a4bc19-2b78-2df4-9cfc-9854e4dbb934}]" testing.go:149: cluster: 2021-10-08T13:01:18.887Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: 2021-10-08T13:01:18.887Z [INFO] server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934: entering follower state: follower="Node at 94a4bc19-2b78-2df4-9cfc-9854e4dbb934 [Follower]" leader= testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [WARN] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering candidate state: node="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Candidate]" term=2 testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.966Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: election won: tally=2 testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: entering leader state: leader="Node at a70513bc-bebd-07bc-2f5f-77d0d59c2e78 [Leader]" testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: added peer, starting replication: peer=server-813c3757-1864-5f0c-0b36-edbd775f2cfd testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: added peer, starting replication: peer=server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: pipelining replication: peer="{Voter server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 94a4bc19-2b78-2df4-9cfc-9854e4dbb934}" testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:18.967Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: pipelining replication: peer="{Voter server-813c3757-1864-5f0c-0b36-edbd775f2cfd 813c3757-1864-5f0c-0b36-edbd775f2cfd}" testing.go:149: cluster: 2021-10-08T13:01:19.072Z [INFO] stable state for Leader reached at 2021-10-08 13:01:18.96710849 +0000 UTC m=+18.981791242 (1 nodes), 79.571984ms from start of poll, 80.086809ms from cluster start. Timeout at 2021-10-08 13:01:19.072199941 +0000 UTC m=+19.086882683, 105.091441ms after stability testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: cannot transfer leadership to itself testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: aborting pipeline replication: peer="{Voter server-94a4bc19-2b78-2df4-9cfc-9854e4dbb934 94a4bc19-2b78-2df4-9cfc-9854e4dbb934}" testing.go:149: server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: 2021-10-08T13:01:19.072Z [INFO] server-a70513bc-bebd-07bc-2f5f-77d0d59c2e78: aborting pipeline replication: peer="{Voter server-813c3757-1864-5f0c-0b36-edbd775f2cfd 813c3757-1864-5f0c-0b36-edbd775f2cfd}" === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) raft_test.go:2064: Disabled unreliable test === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-2132bd59-fcd5-d6c5-055c-d21a18176092: 2021-10-08T13:01:19.074Z [INFO] server-2132bd59-fcd5-d6c5-055c-d21a18176092: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.075Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testLoggerAdapter verbose: server-2e33b5f0-0856-d34c-604e-385dace5e0a0: 2021-10-08T13:01:19.075Z [INFO] server-2e33b5f0-0856-d34c-604e-385dace5e0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.075Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.076Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering follower state: follower="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Follower]" leader= testLoggerAdapter verbose: server-2132bd59-fcd5-d6c5-055c-d21a18176092: 2021-10-08T13:01:19.076Z [INFO] server-2132bd59-fcd5-d6c5-055c-d21a18176092: entering follower state: follower="Node at 2132bd59-fcd5-d6c5-055c-d21a18176092 [Follower]" leader= testLoggerAdapter verbose: server-2e33b5f0-0856-d34c-604e-385dace5e0a0: 2021-10-08T13:01:19.076Z [INFO] server-2e33b5f0-0856-d34c-604e-385dace5e0a0: entering follower state: follower="Node at 2e33b5f0-0856-d34c-604e-385dace5e0a0 [Follower]" leader= testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering candidate state: node="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Candidate]" term=2 testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: election won: tally=2 testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering leader state: leader="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Leader]" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: added peer, starting replication: peer=server-2132bd59-fcd5-d6c5-055c-d21a18176092 testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: added peer, starting replication: peer=server-2e33b5f0-0856-d34c-604e-385dace5e0a0 testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2e33b5f0-0856-d34c-604e-385dace5e0a0 2e33b5f0-0856-d34c-604e-385dace5e0a0}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.235Z [INFO] stable state for Leader reached at 2021-10-08 13:01:19.13064942 +0000 UTC m=+19.145332181 (1 nodes), 54.73965ms from start of poll, 56.159495ms from cluster start. Timeout at 2021-10-08 13:01:19.23572256 +0000 UTC m=+19.250405288, 105.073107ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.235Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.341Z [INFO] stable state for Follower reached at 2021-10-08 13:01:19.235877635 +0000 UTC m=+19.250560355 (2 nodes), 788ns from start of poll, 161.387669ms from cluster start. Timeout at 2021-10-08 13:01:19.340974607 +0000 UTC m=+19.355657327, 105.096972ms after stability testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.476Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.477Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.674Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2132bd59-fcd5-d6c5-055c-d21a18176092 time=84.609827ms testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2132bd59-fcd5-d6c5-055c-d21a18176092 time=86.462004ms testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2e33b5f0-0856-d34c-604e-385dace5e0a0 time=85.980781ms testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering follower state: follower="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Follower]" leader= testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testLoggerAdapter verbose: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2e33b5f0-0856-d34c-604e-385dace5e0a0 2e33b5f0-0856-d34c-604e-385dace5e0a0}" --- FAIL: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.73s) testing.go:149: server-2132bd59-fcd5-d6c5-055c-d21a18176092: 2021-10-08T13:01:19.074Z [INFO] server-2132bd59-fcd5-d6c5-055c-d21a18176092: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.075Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testing.go:149: server-2e33b5f0-0856-d34c-604e-385dace5e0a0: 2021-10-08T13:01:19.075Z [INFO] server-2e33b5f0-0856-d34c-604e-385dace5e0a0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2132bd59-fcd5-d6c5-055c-d21a18176092 Address:2132bd59-fcd5-d6c5-055c-d21a18176092} {Suffrage:Voter ID:server-8758d348-793c-a135-7d51-dc5220e39db4 Address:8758d348-793c-a135-7d51-dc5220e39db4} {Suffrage:Voter ID:server-2e33b5f0-0856-d34c-604e-385dace5e0a0 Address:2e33b5f0-0856-d34c-604e-385dace5e0a0}]" testing.go:149: cluster: 2021-10-08T13:01:19.075Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.076Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering follower state: follower="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Follower]" leader= testing.go:149: server-2132bd59-fcd5-d6c5-055c-d21a18176092: 2021-10-08T13:01:19.076Z [INFO] server-2132bd59-fcd5-d6c5-055c-d21a18176092: entering follower state: follower="Node at 2132bd59-fcd5-d6c5-055c-d21a18176092 [Follower]" leader= testing.go:149: server-2e33b5f0-0856-d34c-604e-385dace5e0a0: 2021-10-08T13:01:19.076Z [INFO] server-2e33b5f0-0856-d34c-604e-385dace5e0a0: entering follower state: follower="Node at 2e33b5f0-0856-d34c-604e-385dace5e0a0 [Follower]" leader= testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering candidate state: node="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Candidate]" term=2 testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: election won: tally=2 testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering leader state: leader="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Leader]" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: added peer, starting replication: peer=server-2132bd59-fcd5-d6c5-055c-d21a18176092 testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: added peer, starting replication: peer=server-2e33b5f0-0856-d34c-604e-385dace5e0a0 testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.130Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2e33b5f0-0856-d34c-604e-385dace5e0a0 2e33b5f0-0856-d34c-604e-385dace5e0a0}" testing.go:149: cluster: 2021-10-08T13:01:19.235Z [INFO] stable state for Leader reached at 2021-10-08 13:01:19.13064942 +0000 UTC m=+19.145332181 (1 nodes), 54.73965ms from start of poll, 56.159495ms from cluster start. Timeout at 2021-10-08 13:01:19.23572256 +0000 UTC m=+19.250405288, 105.073107ms after stability testing.go:149: cluster: 2021-10-08T13:01:19.235Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-08T13:01:19.341Z [INFO] stable state for Follower reached at 2021-10-08 13:01:19.235877635 +0000 UTC m=+19.250560355 (2 nodes), 788ns from start of poll, 161.387669ms from cluster start. Timeout at 2021-10-08 13:01:19.340974607 +0000 UTC m=+19.355657327, 105.096972ms after stability testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.476Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.477Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: pipelining replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.674Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2132bd59-fcd5-d6c5-055c-d21a18176092 time=84.609827ms testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2132bd59-fcd5-d6c5-055c-d21a18176092 time=86.462004ms testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact: server-id=server-2e33b5f0-0856-d34c-604e-385dace5e0a0 time=85.980781ms testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [WARN] server-8758d348-793c-a135-7d51-dc5220e39db4: failed to contact quorum of nodes, stepping down testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: entering follower state: follower="Node at 8758d348-793c-a135-7d51-dc5220e39db4 [Follower]" leader= testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2132bd59-fcd5-d6c5-055c-d21a18176092 2132bd59-fcd5-d6c5-055c-d21a18176092}" testing.go:149: server-8758d348-793c-a135-7d51-dc5220e39db4: 2021-10-08T13:01:19.774Z [INFO] server-8758d348-793c-a135-7d51-dc5220e39db4: aborting pipeline replication: peer="{Voter server-2e33b5f0-0856-d34c-604e-385dace5e0a0 2e33b5f0-0856-d34c-604e-385dace5e0a0}" raft_test.go:2122: leadership transfer should err with: leadership transfer timeout === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.800Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.800Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering follower state: follower="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Follower]" leader= testLoggerAdapter verbose: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:19.800Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.800Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:19.800Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: entering follower state: follower="Node at 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 [Follower]" leader= testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [WARN] server-04191325-3f40-5dea-b5b7-d4464818bfb3: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering candidate state: node="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Candidate]" term=2 testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: election won: tally=2 testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering leader state: leader="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Leader]" testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: added peer, starting replication: peer=server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: pipelining replication: peer="{Voter server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}" testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.995Z [INFO] stable state for Follower reached at 2021-10-08 13:01:19.890821736 +0000 UTC m=+19.905504455 (1 nodes), 90.381785ms from start of poll, 90.826858ms from cluster start. Timeout at 2021-10-08 13:01:19.995897816 +0000 UTC m=+20.010580555, 105.0761ms after stability testLoggerAdapter verbose: cluster: 2021-10-08T13:01:19.996Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-08T13:01:20.101Z [INFO] stable state for Leader reached at 2021-10-08 13:01:19.996090111 +0000 UTC m=+20.010772839 (1 nodes), 1.038µs from start of poll, 196.095242ms from cluster start. Timeout at 2021-10-08 13:01:20.10119096 +0000 UTC m=+20.115873683, 105.100844ms after stability testLoggerAdapter verbose: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:20.111Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testLoggerAdapter verbose: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:20.111Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: aborting pipeline replication: peer="{Voter server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s) testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.800Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.800Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering follower state: follower="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Follower]" leader= testing.go:149: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:19.800Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testing.go:149: cluster: 2021-10-08T13:01:19.800Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:19.800Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: entering follower state: follower="Node at 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 [Follower]" leader= testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [WARN] server-04191325-3f40-5dea-b5b7-d4464818bfb3: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering candidate state: node="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Candidate]" term=2 testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: election won: tally=2 testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: entering leader state: leader="Node at 04191325-3f40-5dea-b5b7-d4464818bfb3 [Leader]" testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: added peer, starting replication: peer=server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:19.890Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: pipelining replication: peer="{Voter server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}" testing.go:149: cluster: 2021-10-08T13:01:19.995Z [INFO] stable state for Follower reached at 2021-10-08 13:01:19.890821736 +0000 UTC m=+19.905504455 (1 nodes), 90.381785ms from start of poll, 90.826858ms from cluster start. Timeout at 2021-10-08 13:01:19.995897816 +0000 UTC m=+20.010580555, 105.0761ms after stability testing.go:149: cluster: 2021-10-08T13:01:19.996Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-08T13:01:20.101Z [INFO] stable state for Leader reached at 2021-10-08 13:01:19.996090111 +0000 UTC m=+20.010772839 (1 nodes), 1.038µs from start of poll, 196.095242ms from cluster start. Timeout at 2021-10-08 13:01:20.10119096 +0000 UTC m=+20.115873683, 105.100844ms after stability testing.go:149: server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: 2021-10-08T13:01:20.111Z [INFO] server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-04191325-3f40-5dea-b5b7-d4464818bfb3 Address:04191325-3f40-5dea-b5b7-d4464818bfb3} {Suffrage:Voter ID:server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 Address:2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}]" testing.go:149: server-04191325-3f40-5dea-b5b7-d4464818bfb3: 2021-10-08T13:01:20.111Z [INFO] server-04191325-3f40-5dea-b5b7-d4464818bfb3: aborting pipeline replication: peer="{Voter server-2bf35aa3-25f9-b32a-21f8-0b7dfad51a97 2bf35aa3-25f9-b32a-21f8-0b7dfad51a97}" === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) FAIL FAIL github.com/hashicorp/raft 20.141s ? github.com/hashicorp/raft/bench [no test files] FAIL dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 make: *** [debian/rules:9: build-indep] Error 25 dpkg-buildpackage: error: debian/rules build-indep subprocess returned exit status 2 E: run_chroot failed: E: command failed: chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all" W: listening on child socket failed: I: removing tempdir /tmp/mmdebstrap.RjNL4jYtQs... mmdebstrap failed