Input buildinfo: https://buildinfos.debian.net/buildinfo-pool/g/golang-github-hashicorp-raft/golang-github-hashicorp-raft_1.1.2-1_all.buildinfo Use metasnap for getting required timestamps New buildinfo file: /tmp/golang-github-hashicorp-raft-1.1.2-1owqhmuej/golang-github-hashicorp-raft_1.1.2-1_all.buildinfo Get source package info: golang-github-hashicorp-raft=1.1.2-1 Source URL: http://snapshot.notset.fr/mr/package/golang-github-hashicorp-raft/1.1.2-1/srcfiles?fileinfo=1 env -i PATH=/usr/sbin:/usr/bin:/sbin:/bin TMPDIR=/tmp mmdebstrap --arch=amd64 --include=autoconf=2.69-11.1 automake=1:1.16.1-4 autopoint=0.19.8.1-10 autotools-dev=20180224.1 base-files=11 base-passwd=3.5.47 bash=5.0-5 binutils=2.33.50.20200115-2 binutils-common=2.33.50.20200115-2 binutils-x86-64-linux-gnu=2.33.50.20200115-2 bsdmainutils=11.1.2+b1 bsdutils=1:2.34-0.1 build-essential=12.8 bzip2=1.0.8-2 coreutils=8.30-3+b1 cpp=4:9.2.1-3.1 cpp-9=9.2.1-24 dash=0.5.10.2-6 debconf=1.5.73 debhelper=12.8 debianutils=4.9.1 dh-autoreconf=19 dh-golang=1.45 dh-strip-nondeterminism=1.6.3-2 diffutils=1:3.7-3 dpkg=1.19.7 dpkg-dev=1.19.7 dwz=0.13-5 fdisk=2.34-0.1 file=1:5.38-4 findutils=4.7.0-1 g++=4:9.2.1-3.1 g++-9=9.2.1-24 gcc=4:9.2.1-3.1 gcc-9=9.2.1-24 gcc-9-base=9.2.1-24 gettext=0.19.8.1-10 gettext-base=0.19.8.1-10 golang-1.13-go=1.13.6-2 golang-1.13-src=1.13.6-2 golang-any=2:1.13~1 golang-github-alecthomas-units-dev=0.0~git20151022.0.2efee85-4 golang-github-armon-go-metrics-dev=0.3.0-1 golang-github-beorn7-perks-dev=1.0.1-1 golang-github-cespare-xxhash-dev=2.1.1-1 golang-github-circonus-labs-circonus-gometrics-dev=2.3.1-2 golang-github-circonus-labs-circonusllhist-dev=0.0~git20160526.0.d724266-2 golang-github-datadog-datadog-go-dev=2.1.0-2 golang-github-davecgh-go-spew-dev=1.1.1-2 golang-github-dgrijalva-jwt-go-v3-dev=3.2.0-2 golang-github-fatih-color-dev=1.7.0-1 golang-github-go-kit-kit-dev=0.6.0-2 golang-github-go-logfmt-logfmt-dev=0.3.0-1 golang-github-go-stack-stack-dev=1.5.2-2 golang-github-golang-mock-dev=1.3.1-2 golang-github-google-go-cmp-dev=0.3.1-1 golang-github-google-gofuzz-dev=0.0~git20170612.24818f7-1 golang-github-hashicorp-go-cleanhttp-dev=0.5.1-1 golang-github-hashicorp-go-hclog-dev=0.11.0-1 golang-github-hashicorp-go-immutable-radix-dev=1.1.0-1 golang-github-hashicorp-go-msgpack-dev=0.5.5-1 golang-github-hashicorp-go-retryablehttp-dev=0.6.4-1 golang-github-hashicorp-go-uuid-dev=1.0.1-2 golang-github-hashicorp-golang-lru-dev=0.5.4-1 golang-github-jpillora-backoff-dev=1.0.0-1 golang-github-json-iterator-go-dev=1.1.4-1 golang-github-julienschmidt-httprouter-dev=1.1-5 golang-github-kr-pretty-dev=0.1.0-1 golang-github-kr-pty-dev=1.1.6-1 golang-github-kr-text-dev=0.1.0-1 golang-github-mattn-go-colorable-dev=0.0.9-3 golang-github-mattn-go-isatty-dev=0.0.8-2 golang-github-modern-go-concurrent-dev=1.0.3-1 golang-github-modern-go-reflect2-dev=1.0.0-1 golang-github-mwitkow-go-conntrack-dev=0.0~git20190716.2f06839-1 golang-github-opentracing-opentracing-go-dev=1.0.2-1 golang-github-pascaldekloe-goe-dev=0.1.0-2 golang-github-pkg-errors-dev=0.8.1-1 golang-github-pmezard-go-difflib-dev=1.0.0-2 golang-github-prometheus-client-golang-dev=1.2.1-3 golang-github-prometheus-client-model-dev=0.0.2+git20171117.99fa1f4-1 golang-github-prometheus-common-dev=0.7.0-1 golang-github-sirupsen-logrus-dev=1.4.2-1 golang-github-stretchr-objx-dev=0.1.1+git20180825.ef50b0d-1 golang-github-stretchr-testify-dev=1.4.0+ds-1 golang-github-tv42-httpunix-dev=0.0~git20150427.b75d861-2 golang-github-ugorji-go-codec-dev=1.1.7-1 golang-github-ugorji-go-msgpack-dev=0.0~git20130605.792643-5 golang-github-vmihailenco-tagparser-dev=0.1.1-2 golang-glog-dev=0.0~git20160126.23def4e-3 golang-go=2:1.13~1 golang-golang-x-net-dev=1:0.0+git20191112.2180aed+dfsg-3 golang-golang-x-oauth2-dev=0.0~git20190604.0f29369-2 golang-golang-x-oauth2-google-dev=0.0~git20190604.0f29369-2 golang-golang-x-sync-dev=0.0~git20190423.1122301-1 golang-golang-x-sys-dev=0.0~git20190726.fc99dfb-1 golang-golang-x-text-dev=0.3.2-3 golang-golang-x-time-dev=0.0~git20161028.0.f51c127-2 golang-golang-x-tools-dev=1:0.0~git20191118.07fc4c7+ds-1 golang-golang-x-xerrors-dev=0.0~git20190717.a985d34-1 golang-google-cloud-compute-metadata-dev=0.43.0-1 golang-google-genproto-dev=0.0~git20190801.fa694d8-2 golang-google-grpc-dev=1.22.1-1 golang-gopkg-alecthomas-kingpin.v2-dev=2.2.6-1 golang-gopkg-check.v1-dev=0.0+git20180628.788fd78-1 golang-gopkg-mgo.v2-dev=2016.08.01-6 golang-gopkg-tomb.v2-dev=0.0~git20161208.d5d1b58-3 golang-gopkg-vmihailenco-msgpack.v2-dev=4.3.1-1 golang-gopkg-yaml.v2-dev=2.2.2-1 golang-goprotobuf-dev=1.3.2-2 golang-procfs-dev=0.0.8-1 golang-prometheus-client-dev=0.9.0-1 golang-protobuf-extensions-dev=1.0.1-1 golang-src=2:1.13~1 grep=3.3-1 groff-base=1.22.4-4 gzip=1.9-3+b1 hostname=3.23 init-system-helpers=1.57 intltool-debian=0.35.0+20060710.5 libacl1=2.2.53-5 libarchive-zip-perl=1.67-1 libasan5=9.2.1-24 libatomic1=9.2.1-24 libattr1=1:2.4.48-5 libaudit-common=1:2.8.5-2 libaudit1=1:2.8.5-2+b1 libbinutils=2.33.50.20200115-2 libblkid1=2.34-0.1 libbsd0=0.10.0-1 libbz2-1.0=1.0.8-2 libc-bin=2.29-9 libc-dev-bin=2.29-9 libc6=2.29-9 libc6-dev=2.29-9 libcap-ng0=0.7.9-2.1+b1 libcc1-0=9.2.1-24 libcroco3=0.6.13-1 libcrypt-dev=1:4.4.10-10 libcrypt1=1:4.4.10-10 libctf-nobfd0=2.33.50.20200115-2 libctf0=2.33.50.20200115-2 libdb5.3=5.3.28+dfsg1-0.6 libdebconfclient0=0.250 libdebhelper-perl=12.8 libdpkg-perl=1.19.7 libelf1=0.176-1.1 libfdisk1=2.34-0.1 libffi7=3.3-3 libfile-stripnondeterminism-perl=1.6.3-2 libgcc-9-dev=9.2.1-24 libgcc1=1:9.2.1-24 libgcrypt20=1.8.5-3 libgdbm-compat4=1.18.1-5 libgdbm6=1.18.1-5 libglib2.0-0=2.62.4-1+b1 libgmp10=2:6.1.2+dfsg-4 libgomp1=9.2.1-24 libgpg-error0=1.36-7 libicu63=63.2-2 libisl22=0.22-2 libitm1=9.2.1-24 liblsan0=9.2.1-24 liblz4-1=1.9.2-2 liblzma5=5.2.4-1+b1 libmagic-mgc=1:5.38-4 libmagic1=1:5.38-4 libmount1=2.34-0.1 libmpc3=1.1.0-1 libmpfr6=4.0.2-1 libncursesw6=6.1+20191019-1 libpam-modules=1.3.1-5 libpam-modules-bin=1.3.1-5 libpam-runtime=1.3.1-5 libpam0g=1.3.1-5 libpcre2-8-0=10.34-7 libpcre3=2:8.39-12+b1 libperl5.30=5.30.0-9 libpipeline1=1.5.2-2 libprotobuf-dev=3.6.1.3-2+b1 libprotobuf-lite17=3.6.1.3-2+b1 libprotobuf17=3.6.1.3-2+b1 libprotoc17=3.6.1.3-2+b1 libquadmath0=9.2.1-24 libsasl2-2=2.1.27+dfsg-2 libsasl2-dev=2.1.27+dfsg-2 libsasl2-modules-db=2.1.27+dfsg-2 libseccomp2=2.4.2-2 libselinux1=3.0-1 libsigsegv2=2.12-2 libsmartcols1=2.34-0.1 libstdc++-9-dev=9.2.1-24 libstdc++6=9.2.1-24 libsub-override-perl=0.09-2 libsystemd0=244-3 libtinfo6=6.1+20191019-1 libtool=2.4.6-11 libtsan0=9.2.1-24 libubsan1=9.2.1-24 libuchardet0=0.0.6-3 libudev1=244-3 libunistring2=0.9.10-2 libuuid1=2.34-0.1 libxml2=2.9.4+dfsg1-8 linux-libc-dev=5.4.13-1 login=1:4.8-1 lsb-base=11.1.0 m4=1.4.18-4 make=4.2.1-1.2 man-db=2.9.0-2 mawk=1.3.3-20 ncurses-base=6.1+20191019-1 ncurses-bin=6.1+20191019-1 patch=2.7.6-6 perl=5.30.0-9 perl-base=5.30.0-9 perl-modules-5.30=5.30.0-9 po-debconf=1.0.21 protobuf-compiler=3.6.1.3-2+b1 sed=4.7-1 sensible-utils=0.0.12+nmu1 sysvinit-utils=2.96-2.1 tar=1.30+dfsg-6+b1 util-linux=2.34-0.1 xz-utils=5.2.4-1+b1 zlib1g=1:1.2.11.dfsg-1+b1 zlib1g-dev=1:1.2.11.dfsg-1+b1 --variant=apt --aptopt=Acquire::Check-Valid-Until "false" --aptopt=Acquire::http::Dl-Limit "1000"; --aptopt=Acquire::https::Dl-Limit "1000"; --aptopt=Acquire::Retries "5"; --aptopt=APT::Get::allow-downgrades "true"; --keyring=/usr/share/keyrings/ --essential-hook=chroot "$1" sh -c "apt-get --yes install fakeroot util-linux" --essential-hook=copy-in /usr/share/keyrings/debian-archive-bullseye-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-security-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-stable.gpg /usr/share/keyrings/debian-archive-buster-automatic.gpg /usr/share/keyrings/debian-archive-buster-security-automatic.gpg /usr/share/keyrings/debian-archive-buster-stable.gpg /usr/share/keyrings/debian-archive-keyring.gpg /usr/share/keyrings/debian-archive-removed-keys.gpg /usr/share/keyrings/debian-archive-stretch-automatic.gpg /usr/share/keyrings/debian-archive-stretch-security-automatic.gpg /usr/share/keyrings/debian-archive-stretch-stable.gpg /usr/share/keyrings/debian-ports-archive-keyring-removed.gpg /usr/share/keyrings/debian-ports-archive-keyring.gpg /usr/share/keyrings/debian-keyring.gpg /etc/apt/trusted.gpg.d/ --essential-hook=chroot "$1" sh -c "rm /etc/apt/sources.list && echo 'deb http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb-src http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb http://snapshot.notset.fr/archive/debian/20200122T091601Z/ unstable main' >> /etc/apt/sources.list && apt-get update" --customize-hook=chroot "$1" useradd --no-create-home -d /nonexistent -p "" builduser -s /bin/bash --customize-hook=chroot "$1" env sh -c "apt-get source --only-source -d golang-github-hashicorp-raft=1.1.2-1 && mkdir -p /build/golang-github-hashicorp-raft-HJm3EX && dpkg-source --no-check -x /*.dsc /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && chown -R builduser:builduser /build/golang-github-hashicorp-raft-HJm3EX" --customize-hook=chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all" --customize-hook=sync-out /build/golang-github-hashicorp-raft-HJm3EX /tmp/golang-github-hashicorp-raft-1.1.2-1owqhmuej bullseye /dev/null deb http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable main I: automatically chosen mode: root I: chroot architecture amd64 is equal to the host's architecture I: automatically chosen format: tar I: using /tmp/mmdebstrap.ZLm5cCJwWc as tempdir I: running apt-get update... I: downloading packages with apt... I: extracting archives... I: installing essential packages... I: running --essential-hook in shell: sh -c 'chroot "$1" sh -c "apt-get --yes install fakeroot util-linux"' exec /tmp/mmdebstrap.ZLm5cCJwWc Reading package lists... Building dependency tree... util-linux is already the newest version (2.34-0.1). The following NEW packages will be installed: fakeroot libfakeroot 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 132 kB of archives. After this operation, 393 kB of additional disk space will be used. Get:1 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable/main amd64 libfakeroot amd64 1.24-1 [45.7 kB] Get:2 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable/main amd64 fakeroot amd64 1.24-1 [85.9 kB] debconf: delaying package configuration, since apt-utils is not installed Fetched 132 kB in 0s (754 kB/s) Selecting previously unselected package libfakeroot:amd64. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 4508 files and directories currently installed.) Preparing to unpack .../libfakeroot_1.24-1_amd64.deb ... Unpacking libfakeroot:amd64 (1.24-1) ... Selecting previously unselected package fakeroot. Preparing to unpack .../fakeroot_1.24-1_amd64.deb ... Unpacking fakeroot (1.24-1) ... Setting up libfakeroot:amd64 (1.24-1) ... Setting up fakeroot (1.24-1) ... update-alternatives: using /usr/bin/fakeroot-sysv to provide /usr/bin/fakeroot (fakeroot) in auto mode Processing triggers for libc-bin (2.29-9) ... I: running special hook: copy-in /usr/share/keyrings/debian-archive-bullseye-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-security-automatic.gpg /usr/share/keyrings/debian-archive-bullseye-stable.gpg /usr/share/keyrings/debian-archive-buster-automatic.gpg /usr/share/keyrings/debian-archive-buster-security-automatic.gpg /usr/share/keyrings/debian-archive-buster-stable.gpg /usr/share/keyrings/debian-archive-keyring.gpg /usr/share/keyrings/debian-archive-removed-keys.gpg /usr/share/keyrings/debian-archive-stretch-automatic.gpg /usr/share/keyrings/debian-archive-stretch-security-automatic.gpg /usr/share/keyrings/debian-archive-stretch-stable.gpg /usr/share/keyrings/debian-ports-archive-keyring-removed.gpg /usr/share/keyrings/debian-ports-archive-keyring.gpg /usr/share/keyrings/debian-keyring.gpg /etc/apt/trusted.gpg.d/ I: running --essential-hook in shell: sh -c 'chroot "$1" sh -c "rm /etc/apt/sources.list && echo 'deb http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb-src http://snapshot.notset.fr/archive/debian/20210814T212851Z/ bookworm main deb http://snapshot.notset.fr/archive/debian/20200122T091601Z/ unstable main' >> /etc/apt/sources.list && apt-get update"' exec /tmp/mmdebstrap.ZLm5cCJwWc Get:1 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm InRelease [81.6 kB] Hit:2 http://snapshot.notset.fr/archive/debian/20200122T091601Z unstable InRelease Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Ign:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources Ign:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages Get:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main Sources [11.4 MB] Get:4 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main amd64 Packages [11.1 MB] Fetched 22.6 MB in 21s (1053 kB/s) Reading package lists... I: installing remaining packages inside the chroot... I: running --customize-hook in shell: sh -c 'chroot "$1" useradd --no-create-home -d /nonexistent -p "" builduser -s /bin/bash' exec /tmp/mmdebstrap.ZLm5cCJwWc I: running --customize-hook in shell: sh -c 'chroot "$1" env sh -c "apt-get source --only-source -d golang-github-hashicorp-raft=1.1.2-1 && mkdir -p /build/golang-github-hashicorp-raft-HJm3EX && dpkg-source --no-check -x /*.dsc /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && chown -R builduser:builduser /build/golang-github-hashicorp-raft-HJm3EX"' exec /tmp/mmdebstrap.ZLm5cCJwWc Reading package lists... NOTICE: 'golang-github-hashicorp-raft' packaging is maintained in the 'Git' version control system at: https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git Please use: git clone https://salsa.debian.org/go-team/packages/golang-github-hashicorp-raft.git to retrieve the latest (possibly unreleased) updates to the package. Need to get 115 kB of source archives. Get:1 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (dsc) [2408 B] Get:2 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (tar) [103 kB] Get:3 http://snapshot.notset.fr/archive/debian/20210814T212851Z bookworm/main golang-github-hashicorp-raft 1.1.2-1 (diff) [8980 B] Fetched 115 kB in 0s (472 kB/s) Download complete and in download only mode W: Download is performed unsandboxed as root as file 'golang-github-hashicorp-raft_1.1.2-1.dsc' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied) dpkg-source: info: extracting golang-github-hashicorp-raft in /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2.orig.tar.xz dpkg-source: info: unpacking golang-github-hashicorp-raft_1.1.2-1.debian.tar.xz dpkg-source: info: using patch list from debian/patches/series dpkg-source: info: applying t-skip--TestNetworkTransport_AppendEntriesPipeline_CloseStreams.patch dpkg-source: info: applying t-skip--TestRaft_AutoSnapshot.patch dpkg-source: info: applying t-skip--TestRaft_LeadershipTransferLeaderRejectsClientRequests.patch dpkg-source: info: applying t-skip--TestRaft_RecoverCluster.patch dpkg-source: info: applying t-skip--TestRaft_SnapshotRestore_PeerChange.patch I: running --customize-hook in shell: sh -c 'chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all"' exec /tmp/mmdebstrap.ZLm5cCJwWc dpkg-buildpackage: info: source package golang-github-hashicorp-raft dpkg-buildpackage: info: source version 1.1.2-1 dpkg-buildpackage: info: source distribution unstable dpkg-buildpackage: info: source changed by Dmitry Smirnov dpkg-source --before-build . fakeroot debian/rules clean dh clean --buildsystem=golang --with=golang dh_auto_clean -O--buildsystem=golang dh_autoreconf_clean -O--buildsystem=golang dh_clean -O--buildsystem=golang debian/rules build-indep dh build-indep --buildsystem=golang --with=golang dh_update_autotools_config -i -O--buildsystem=golang dh_autoreconf -i -O--buildsystem=golang dh_auto_configure -i -O--buildsystem=golang dh_auto_build -i -O--buildsystem=golang cd obj-x86_64-linux-gnu && go install -trimpath -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench internal/race runtime/internal/sys internal/cpu runtime/internal/atomic runtime/internal/math sync/atomic unicode unicode/utf8 container/list crypto/internal/subtle internal/bytealg crypto/subtle math/bits internal/testlog math encoding unicode/utf16 internal/nettrace vendor/golang.org/x/crypto/cryptobyte/asn1 runtime/cgo runtime vendor/golang.org/x/crypto/internal/subtle vendor/golang.org/x/crypto/curve25519 internal/reflectlite sync internal/singleflight math/rand sort errors io internal/oserror strconv syscall github.com/hashicorp/golang-lru/simplelru vendor/golang.org/x/net/dns/dnsmessage bytes reflect bufio strings hash hash/crc32 time internal/syscall/unix crypto github.com/hashicorp/go-immutable-radix crypto/internal/randutil crypto/hmac crypto/rc4 vendor/golang.org/x/crypto/hkdf vendor/golang.org/x/text/transform path context internal/poll regexp/syntax html encoding/binary internal/fmtsort os crypto/cipher encoding/base64 crypto/sha512 crypto/ed25519/internal/edwards25519 crypto/aes crypto/des fmt net crypto/md5 crypto/sha1 crypto/sha256 encoding/pem math/big encoding/json log compress/flate compress/gzip encoding/hex path/filepath net/url io/ioutil crypto/rand crypto/elliptic encoding/asn1 crypto/ed25519 crypto/rsa crypto/dsa crypto/ecdsa crypto/x509/pkix vendor/golang.org/x/crypto/cryptobyte vendor/golang.org/x/crypto/internal/chacha20 vendor/golang.org/x/crypto/poly1305 vendor/golang.org/x/sys/cpu vendor/golang.org/x/text/unicode/bidi vendor/golang.org/x/crypto/chacha20poly1305 vendor/golang.org/x/text/unicode/norm vendor/golang.org/x/text/secure/bidirule vendor/golang.org/x/net/http2/hpack mime mime/quotedprintable net/http/internal os/signal vendor/golang.org/x/net/idna golang.org/x/sys/unix regexp encoding/gob go/token crypto/x509 net/textproto vendor/golang.org/x/net/http/httpproxy vendor/golang.org/x/net/http/httpguts mime/multipart github.com/mattn/go-isatty github.com/mattn/go-colorable text/template/parse crypto/tls github.com/fatih/color hash/crc64 github.com/hashicorp/go-hclog flag runtime/debug runtime/trace text/template testing html/template net/http/httptrace net/http net/rpc github.com/armon/go-metrics github.com/hashicorp/go-msgpack/codec github.com/hashicorp/raft github.com/hashicorp/raft/bench dh_auto_test -i -O--buildsystem=golang cd obj-x86_64-linux-gnu && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench === RUN TestCommitment_setVoters --- PASS: TestCommitment_setVoters (0.00s) === RUN TestCommitment_match_max --- PASS: TestCommitment_match_max (0.00s) === RUN TestCommitment_match_nonVoting --- PASS: TestCommitment_match_nonVoting (0.00s) === RUN TestCommitment_recalculate --- PASS: TestCommitment_recalculate (0.00s) === RUN TestCommitment_recalculate_startIndex --- PASS: TestCommitment_recalculate_startIndex (0.00s) === RUN TestCommitment_noVoterSanity --- PASS: TestCommitment_noVoterSanity (0.00s) === RUN TestCommitment_singleVoter --- PASS: TestCommitment_singleVoter (0.00s) === RUN TestConfiguration_Configuration_Clone --- PASS: TestConfiguration_Configuration_Clone (0.00s) === RUN TestConfiguration_configurations_Clone --- PASS: TestConfiguration_configurations_Clone (0.00s) === RUN TestConfiguration_hasVote --- PASS: TestConfiguration_hasVote (0.00s) === RUN TestConfiguration_checkConfiguration --- PASS: TestConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_nextConfiguration_table --- PASS: TestConfiguration_nextConfiguration_table (0.00s) === RUN TestConfiguration_nextConfiguration_prevIndex --- PASS: TestConfiguration_nextConfiguration_prevIndex (0.00s) === RUN TestConfiguration_nextConfiguration_checkConfiguration --- PASS: TestConfiguration_nextConfiguration_checkConfiguration (0.00s) === RUN TestConfiguration_encodeDecodePeers --- PASS: TestConfiguration_encodeDecodePeers (0.00s) === RUN TestConfiguration_encodeDecodeConfiguration --- PASS: TestConfiguration_encodeDecodeConfiguration (0.00s) === RUN TestDiscardSnapshotStoreImpl --- PASS: TestDiscardSnapshotStoreImpl (0.00s) === RUN TestDiscardSnapshotSinkImpl --- PASS: TestDiscardSnapshotSinkImpl (0.00s) === RUN TestFileSnapshotStoreImpl --- PASS: TestFileSnapshotStoreImpl (0.00s) === RUN TestFileSnapshotSinkImpl --- PASS: TestFileSnapshotSinkImpl (0.00s) === RUN TestFileSS_CreateSnapshotMissingParentDir --- PASS: TestFileSS_CreateSnapshotMissingParentDir (0.01s) testing.go:153: 2021-10-19T21:54:18.880Z [INFO] creating new snapshot: path=/tmp/raft560695273/raft423275572/snapshots/3-10-1634680458880.tmp === RUN TestFileSS_CreateSnapshot --- PASS: TestFileSS_CreateSnapshot (0.00s) testing.go:153: 2021-10-19T21:54:18.886Z [INFO] creating new snapshot: path=/tmp/raft594717699/snapshots/3-10-1634680458886.tmp testing.go:153: 2021-10-19T21:54:18.887Z [WARN] found temporary snapshot: name=3-10-1634680458886.tmp === RUN TestFileSS_CancelSnapshot --- PASS: TestFileSS_CancelSnapshot (0.00s) testing.go:153: 2021-10-19T21:54:18.888Z [INFO] creating new snapshot: path=/tmp/raft249580934/snapshots/3-10-1634680458888.tmp === RUN TestFileSS_Retention --- PASS: TestFileSS_Retention (0.01s) testing.go:153: 2021-10-19T21:54:18.890Z [INFO] creating new snapshot: path=/tmp/raft380398893/snapshots/3-10-1634680458890.tmp testing.go:153: 2021-10-19T21:54:18.892Z [INFO] creating new snapshot: path=/tmp/raft380398893/snapshots/3-11-1634680458892.tmp testing.go:153: 2021-10-19T21:54:18.894Z [INFO] creating new snapshot: path=/tmp/raft380398893/snapshots/3-12-1634680458894.tmp testing.go:153: 2021-10-19T21:54:18.896Z [INFO] reaping snapshot: path=/tmp/raft380398893/snapshots/3-10-1634680458890 testing.go:153: 2021-10-19T21:54:18.896Z [INFO] creating new snapshot: path=/tmp/raft380398893/snapshots/3-13-1634680458896.tmp testing.go:153: 2021-10-19T21:54:18.897Z [INFO] reaping snapshot: path=/tmp/raft380398893/snapshots/3-11-1634680458892 testing.go:153: 2021-10-19T21:54:18.897Z [INFO] creating new snapshot: path=/tmp/raft380398893/snapshots/3-14-1634680458897.tmp testing.go:153: 2021-10-19T21:54:18.899Z [INFO] reaping snapshot: path=/tmp/raft380398893/snapshots/3-12-1634680458894 === RUN TestFileSS_BadPerm --- PASS: TestFileSS_BadPerm (0.00s) === RUN TestFileSS_MissingParentDir --- PASS: TestFileSS_MissingParentDir (0.00s) === RUN TestFileSS_Ordering --- PASS: TestFileSS_Ordering (0.00s) testing.go:153: 2021-10-19T21:54:18.900Z [INFO] creating new snapshot: path=/tmp/raft563532380/snapshots/5-130350-1634680458900.tmp testing.go:153: 2021-10-19T21:54:18.902Z [INFO] creating new snapshot: path=/tmp/raft563532380/snapshots/36-204917-1634680458902.tmp === RUN TestDeferFutureSuccess --- PASS: TestDeferFutureSuccess (0.00s) === RUN TestDeferFutureError --- PASS: TestDeferFutureError (0.00s) === RUN TestDeferFutureConcurrent --- PASS: TestDeferFutureConcurrent (0.00s) === RUN TestInmemSnapshotStoreImpl --- PASS: TestInmemSnapshotStoreImpl (0.00s) === RUN TestInmemSnapshotSinkImpl --- PASS: TestInmemSnapshotSinkImpl (0.00s) === RUN TestInmemSS_CreateSnapshot --- PASS: TestInmemSS_CreateSnapshot (0.00s) === RUN TestInmemSS_OpenSnapshotTwice --- PASS: TestInmemSS_OpenSnapshotTwice (0.00s) === RUN TestInmemTransportImpl --- PASS: TestInmemTransportImpl (0.00s) === RUN TestInmemTransportWriteTimeout --- PASS: TestInmemTransportWriteTimeout (0.01s) === RUN TestRaft_Integ --- SKIP: TestRaft_Integ (0.00s) === RUN TestLogCache --- PASS: TestLogCache (0.00s) === RUN TestNetworkTransport_CloseStreams --- PASS: TestNetworkTransport_CloseStreams (0.00s) === RUN TestNetworkTransport_StartStop --- PASS: TestNetworkTransport_StartStop (0.00s) === RUN TestNetworkTransport_Heartbeat_FastPath --- PASS: TestNetworkTransport_Heartbeat_FastPath (0.00s) === RUN TestNetworkTransport_AppendEntries --- PASS: TestNetworkTransport_AppendEntries (0.00s) === RUN TestNetworkTransport_AppendEntriesPipeline --- PASS: TestNetworkTransport_AppendEntriesPipeline (0.02s) === RUN TestNetworkTransport_AppendEntriesPipeline_CloseStreams --- SKIP: TestNetworkTransport_AppendEntriesPipeline_CloseStreams (0.00s) net_transport_test.go:339: Disabled unreliable test === RUN TestNetworkTransport_RequestVote --- PASS: TestNetworkTransport_RequestVote (0.00s) === RUN TestNetworkTransport_InstallSnapshot --- PASS: TestNetworkTransport_InstallSnapshot (0.00s) === RUN TestNetworkTransport_EncodeDecode --- PASS: TestNetworkTransport_EncodeDecode (0.00s) === RUN TestNetworkTransport_EncodeDecode_AddressProvider --- PASS: TestNetworkTransport_EncodeDecode_AddressProvider (0.00s) === RUN TestNetworkTransport_PooledConn --- PASS: TestNetworkTransport_PooledConn (0.00s) === RUN TestNetworkTransport_ListenBackoff --- PASS: TestNetworkTransport_ListenBackoff (4.00s) net_transport_test.go:718: countingWriter: 2021-10-19T21:54:18.946Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:18.951Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:18.961Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:18.981Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:19.021Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:19.102Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:19.262Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:19.582Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:20.222Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:21.223Z [ERROR] test: failed to accept connection: error="intentional error in test" net_transport_test.go:718: countingWriter: 2021-10-19T21:54:22.223Z [ERROR] test: failed to accept connection: error="intentional error in test" === RUN TestPeersJSON_BadConfiguration --- PASS: TestPeersJSON_BadConfiguration (0.00s) === RUN TestPeersJSON_ReadPeersJSON --- PASS: TestPeersJSON_ReadPeersJSON (0.00s) === RUN TestPeersJSON_ReadConfigJSON --- PASS: TestPeersJSON_ReadConfigJSON (0.00s) === RUN TestRaft_StartStop testLoggerAdapter verbose: server-d4abaa29-1133-1752-d5d9-04781a24226a: 2021-10-19T21:54:22.949Z [INFO] server-d4abaa29-1133-1752-d5d9-04781a24226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4abaa29-1133-1752-d5d9-04781a24226a Address:d4abaa29-1133-1752-d5d9-04781a24226a}]" --- PASS: TestRaft_StartStop (0.00s) testing.go:149: server-d4abaa29-1133-1752-d5d9-04781a24226a: 2021-10-19T21:54:22.949Z [INFO] server-d4abaa29-1133-1752-d5d9-04781a24226a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d4abaa29-1133-1752-d5d9-04781a24226a Address:d4abaa29-1133-1752-d5d9-04781a24226a}]" === RUN TestRaft_AfterShutdown testLoggerAdapter verbose: server-c320f356-6106-bb31-185b-c9828178389c: 2021-10-19T21:54:22.950Z [INFO] server-c320f356-6106-bb31-185b-c9828178389c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c320f356-6106-bb31-185b-c9828178389c Address:c320f356-6106-bb31-185b-c9828178389c}]" --- PASS: TestRaft_AfterShutdown (0.00s) testing.go:149: server-c320f356-6106-bb31-185b-c9828178389c: 2021-10-19T21:54:22.950Z [INFO] server-c320f356-6106-bb31-185b-c9828178389c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c320f356-6106-bb31-185b-c9828178389c Address:c320f356-6106-bb31-185b-c9828178389c}]" === RUN TestRaft_LiveBootstrap testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:22.952Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:22.952Z [INFO] server-83330054-b528-b657-00cb-d432324fcc29: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:22.952Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering follower state: follower="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Follower]" leader= testLoggerAdapter verbose: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:22.952Z [INFO] server-83330054-b528-b657-00cb-d432324fcc29: entering follower state: follower="Node at 83330054-b528-b657-00cb-d432324fcc29 [Follower]" leader= testLoggerAdapter verbose: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:22.952Z [INFO] server-12070bc9-9142-b4de-9395-0bc8207eccd5: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:22.952Z [INFO] server-12070bc9-9142-b4de-9395-0bc8207eccd5: entering follower state: follower="Node at 12070bc9-9142-b4de-9395-0bc8207eccd5 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-19T21:54:22.952Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:23.007Z [WARN] server-83330054-b528-b657-00cb-d432324fcc29: no known peers, aborting election testLoggerAdapter verbose: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:23.035Z [WARN] server-12070bc9-9142-b4de-9395-0bc8207eccd5: no known peers, aborting election testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.046Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.046Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering candidate state: node="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Candidate]" term=2 testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: election won: tally=2 testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering leader state: leader="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Leader]" testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: added peer, starting replication: peer=server-83330054-b528-b657-00cb-d432324fcc29 testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: added peer, starting replication: peer=server-12070bc9-9142-b4de-9395-0bc8207eccd5 testLoggerAdapter verbose: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:23.047Z [WARN] server-12070bc9-9142-b4de-9395-0bc8207eccd5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: appendEntries rejected, sending older logs: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" next=1 testLoggerAdapter verbose: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:23.047Z [WARN] server-83330054-b528-b657-00cb-d432324fcc29: failed to get previous log: previous-index=1 last-index=0 error="log not found" testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: pipelining replication: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: appendEntries rejected, sending older logs: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" next=1 testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.048Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: pipelining replication: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.152Z [INFO] stable state for Follower reached at 2021-10-19 21:54:23.047430204 +0000 UTC m=+4.172456388 (2 nodes), 94.431398ms from start of poll, 95.133418ms from cluster start. Timeout at 2021-10-19 21:54:23.152508874 +0000 UTC m=+4.277535021, 105.078633ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.152Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.257Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.15283441 +0000 UTC m=+4.277860538 (1 nodes), 2.859µs from start of poll, 200.537568ms from cluster start. Timeout at 2021-10-19 21:54:23.257911932 +0000 UTC m=+4.382938066, 105.077528ms after stability testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.268Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: aborting pipeline replication: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" testLoggerAdapter verbose: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.268Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: aborting pipeline replication: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" --- PASS: TestRaft_LiveBootstrap (0.32s) testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:22.952Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: initial configuration: index=0 servers=[] testing.go:149: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:22.952Z [INFO] server-83330054-b528-b657-00cb-d432324fcc29: initial configuration: index=0 servers=[] testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:22.952Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering follower state: follower="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Follower]" leader= testing.go:149: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:22.952Z [INFO] server-83330054-b528-b657-00cb-d432324fcc29: entering follower state: follower="Node at 83330054-b528-b657-00cb-d432324fcc29 [Follower]" leader= testing.go:149: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:22.952Z [INFO] server-12070bc9-9142-b4de-9395-0bc8207eccd5: initial configuration: index=0 servers=[] testing.go:149: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:22.952Z [INFO] server-12070bc9-9142-b4de-9395-0bc8207eccd5: entering follower state: follower="Node at 12070bc9-9142-b4de-9395-0bc8207eccd5 [Follower]" leader= testing.go:149: cluster: 2021-10-19T21:54:22.952Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:23.007Z [WARN] server-83330054-b528-b657-00cb-d432324fcc29: no known peers, aborting election testing.go:149: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:23.035Z [WARN] server-12070bc9-9142-b4de-9395-0bc8207eccd5: no known peers, aborting election testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.046Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.046Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering candidate state: node="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Candidate]" term=2 testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: election won: tally=2 testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: entering leader state: leader="Node at b5dd4dee-b95d-8a06-e77b-509278428233 [Leader]" testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: added peer, starting replication: peer=server-83330054-b528-b657-00cb-d432324fcc29 testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: added peer, starting replication: peer=server-12070bc9-9142-b4de-9395-0bc8207eccd5 testing.go:149: server-12070bc9-9142-b4de-9395-0bc8207eccd5: 2021-10-19T21:54:23.047Z [WARN] server-12070bc9-9142-b4de-9395-0bc8207eccd5: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: appendEntries rejected, sending older logs: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" next=1 testing.go:149: server-83330054-b528-b657-00cb-d432324fcc29: 2021-10-19T21:54:23.047Z [WARN] server-83330054-b528-b657-00cb-d432324fcc29: failed to get previous log: previous-index=1 last-index=0 error="log not found" testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: pipelining replication: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.047Z [WARN] server-b5dd4dee-b95d-8a06-e77b-509278428233: appendEntries rejected, sending older logs: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" next=1 testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.048Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: pipelining replication: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" testing.go:149: cluster: 2021-10-19T21:54:23.152Z [INFO] stable state for Follower reached at 2021-10-19 21:54:23.047430204 +0000 UTC m=+4.172456388 (2 nodes), 94.431398ms from start of poll, 95.133418ms from cluster start. Timeout at 2021-10-19 21:54:23.152508874 +0000 UTC m=+4.277535021, 105.078633ms after stability testing.go:149: cluster: 2021-10-19T21:54:23.152Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:23.257Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.15283441 +0000 UTC m=+4.277860538 (1 nodes), 2.859µs from start of poll, 200.537568ms from cluster start. Timeout at 2021-10-19 21:54:23.257911932 +0000 UTC m=+4.382938066, 105.077528ms after stability testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.268Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: aborting pipeline replication: peer="{Voter server-12070bc9-9142-b4de-9395-0bc8207eccd5 12070bc9-9142-b4de-9395-0bc8207eccd5}" testing.go:149: server-b5dd4dee-b95d-8a06-e77b-509278428233: 2021-10-19T21:54:23.268Z [INFO] server-b5dd4dee-b95d-8a06-e77b-509278428233: aborting pipeline replication: peer="{Voter server-83330054-b528-b657-00cb-d432324fcc29 83330054-b528-b657-00cb-d432324fcc29}" === RUN TestRaft_RecoverCluster_NoState testLoggerAdapter verbose: server-97e13c23-5763-595b-351b-7cc6cf3820c5: 2021-10-19T21:54:23.269Z [INFO] server-97e13c23-5763-595b-351b-7cc6cf3820c5: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-97e13c23-5763-595b-351b-7cc6cf3820c5: 2021-10-19T21:54:23.269Z [INFO] server-97e13c23-5763-595b-351b-7cc6cf3820c5: entering follower state: follower="Node at 97e13c23-5763-595b-351b-7cc6cf3820c5 [Shutdown]" leader= --- PASS: TestRaft_RecoverCluster_NoState (0.00s) testing.go:149: server-97e13c23-5763-595b-351b-7cc6cf3820c5: 2021-10-19T21:54:23.269Z [INFO] server-97e13c23-5763-595b-351b-7cc6cf3820c5: initial configuration: index=0 servers=[] testing.go:149: server-97e13c23-5763-595b-351b-7cc6cf3820c5: 2021-10-19T21:54:23.269Z [INFO] server-97e13c23-5763-595b-351b-7cc6cf3820c5: entering follower state: follower="Node at 97e13c23-5763-595b-351b-7cc6cf3820c5 [Shutdown]" leader= === RUN TestRaft_RecoverCluster --- SKIP: TestRaft_RecoverCluster (0.00s) raft_test.go:119: Disabled unreliable test === RUN TestRaft_HasExistingState testLoggerAdapter verbose: server-514236ee-0067-2046-3fef-a8667fecf8ca: 2021-10-19T21:54:23.271Z [INFO] server-514236ee-0067-2046-3fef-a8667fecf8ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2}]" testLoggerAdapter verbose: server-514236ee-0067-2046-3fef-a8667fecf8ca: 2021-10-19T21:54:23.271Z [INFO] server-514236ee-0067-2046-3fef-a8667fecf8ca: entering follower state: follower="Node at 514236ee-0067-2046-3fef-a8667fecf8ca [Follower]" leader= testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.271Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2}]" testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.271Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering follower state: follower="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Follower]" leader= testLoggerAdapter verbose: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.271Z [INFO] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.271Z [INFO] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: entering follower state: follower="Node at b2a7e27b-00ef-b9b9-865f-4f2a7df7703b [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.271Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [WARN] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering candidate state: node="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Candidate]" term=2 testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: election won: tally=2 testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering leader state: leader="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Leader]" testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: added peer, starting replication: peer=server-514236ee-0067-2046-3fef-a8667fecf8ca testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: pipelining replication: peer="{Voter server-514236ee-0067-2046-3fef-a8667fecf8ca 514236ee-0067-2046-3fef-a8667fecf8ca}" testLoggerAdapter verbose: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.354Z [WARN] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.440Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.335183921 +0000 UTC m=+4.460210126 (1 nodes), 63.338926ms from start of poll, 64.138348ms from cluster start. Timeout at 2021-10-19 21:54:23.440270146 +0000 UTC m=+4.565296287, 105.086161ms after stability testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: updating configuration: command=AddStaging server-id=server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b server-addr=b2a7e27b-00ef-b9b9-865f-4f2a7df7703b servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2} {Suffrage:Voter ID:server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b Address:b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}]" testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: added peer, starting replication: peer=server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b testLoggerAdapter verbose: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.440Z [WARN] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [WARN] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: appendEntries rejected, sending older logs: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" next=1 testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.441Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: pipelining replication: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.446Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.551Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.446324463 +0000 UTC m=+4.571350644 (1 nodes), 1.218µs from start of poll, 175.278866ms from cluster start. Timeout at 2021-10-19 21:54:23.551415096 +0000 UTC m=+4.676441297, 105.090653ms after stability testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.551Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: aborting pipeline replication: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" testLoggerAdapter verbose: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.551Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: aborting pipeline replication: peer="{Voter server-514236ee-0067-2046-3fef-a8667fecf8ca 514236ee-0067-2046-3fef-a8667fecf8ca}" --- PASS: TestRaft_HasExistingState (0.28s) testing.go:149: server-514236ee-0067-2046-3fef-a8667fecf8ca: 2021-10-19T21:54:23.271Z [INFO] server-514236ee-0067-2046-3fef-a8667fecf8ca: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2}]" testing.go:149: server-514236ee-0067-2046-3fef-a8667fecf8ca: 2021-10-19T21:54:23.271Z [INFO] server-514236ee-0067-2046-3fef-a8667fecf8ca: entering follower state: follower="Node at 514236ee-0067-2046-3fef-a8667fecf8ca [Follower]" leader= testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.271Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2}]" testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.271Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering follower state: follower="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Follower]" leader= testing.go:149: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.271Z [INFO] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: initial configuration: index=0 servers=[] testing.go:149: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.271Z [INFO] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: entering follower state: follower="Node at b2a7e27b-00ef-b9b9-865f-4f2a7df7703b [Follower]" leader= testing.go:149: cluster: 2021-10-19T21:54:23.271Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [WARN] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering candidate state: node="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Candidate]" term=2 testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.334Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: election won: tally=2 testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: entering leader state: leader="Node at d455eb9b-4f6a-a244-fbfb-0424d17996e2 [Leader]" testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: added peer, starting replication: peer=server-514236ee-0067-2046-3fef-a8667fecf8ca testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.335Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: pipelining replication: peer="{Voter server-514236ee-0067-2046-3fef-a8667fecf8ca 514236ee-0067-2046-3fef-a8667fecf8ca}" testing.go:149: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.354Z [WARN] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: no known peers, aborting election testing.go:149: cluster: 2021-10-19T21:54:23.440Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.335183921 +0000 UTC m=+4.460210126 (1 nodes), 63.338926ms from start of poll, 64.138348ms from cluster start. Timeout at 2021-10-19 21:54:23.440270146 +0000 UTC m=+4.565296287, 105.086161ms after stability testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: updating configuration: command=AddStaging server-id=server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b server-addr=b2a7e27b-00ef-b9b9-865f-4f2a7df7703b servers="[{Suffrage:Voter ID:server-514236ee-0067-2046-3fef-a8667fecf8ca Address:514236ee-0067-2046-3fef-a8667fecf8ca} {Suffrage:Voter ID:server-d455eb9b-4f6a-a244-fbfb-0424d17996e2 Address:d455eb9b-4f6a-a244-fbfb-0424d17996e2} {Suffrage:Voter ID:server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b Address:b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}]" testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: added peer, starting replication: peer=server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b testing.go:149: server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: 2021-10-19T21:54:23.440Z [WARN] server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.440Z [WARN] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: appendEntries rejected, sending older logs: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" next=1 testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.441Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: pipelining replication: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" testing.go:149: cluster: 2021-10-19T21:54:23.446Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:23.551Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.446324463 +0000 UTC m=+4.571350644 (1 nodes), 1.218µs from start of poll, 175.278866ms from cluster start. Timeout at 2021-10-19 21:54:23.551415096 +0000 UTC m=+4.676441297, 105.090653ms after stability testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.551Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: aborting pipeline replication: peer="{Voter server-b2a7e27b-00ef-b9b9-865f-4f2a7df7703b b2a7e27b-00ef-b9b9-865f-4f2a7df7703b}" testing.go:149: server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: 2021-10-19T21:54:23.551Z [INFO] server-d455eb9b-4f6a-a244-fbfb-0424d17996e2: aborting pipeline replication: peer="{Voter server-514236ee-0067-2046-3fef-a8667fecf8ca 514236ee-0067-2046-3fef-a8667fecf8ca}" === RUN TestRaft_SingleNode testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.554Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 Address:8c48f0cc-22ae-ab48-c17c-2f7b31eb3880}]" testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.555Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering follower state: follower="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Follower]" leader= testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [WARN] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering candidate state: node="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Candidate]" term=2 testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: election won: tally=1 testLoggerAdapter verbose: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering leader state: leader="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Leader]" --- PASS: TestRaft_SingleNode (0.08s) testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.554Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 Address:8c48f0cc-22ae-ab48-c17c-2f7b31eb3880}]" testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.555Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering follower state: follower="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Follower]" leader= testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [WARN] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering candidate state: node="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Candidate]" term=2 testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: election won: tally=1 testing.go:149: server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: 2021-10-19T21:54:23.628Z [INFO] server-8c48f0cc-22ae-ab48-c17c-2f7b31eb3880: entering leader state: leader="Node at 8c48f0cc-22ae-ab48-c17c-2f7b31eb3880 [Leader]" === RUN TestRaft_TripleNode testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.633Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.633Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.633Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering follower state: follower="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Follower]" leader= testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.633Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering follower state: follower="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Follower]" leader= testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.633Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.634Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.634Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering follower state: follower="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Follower]" leader= testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.715Z [WARN] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.715Z [WARN] server-5605632c-44b1-3682-0307-b526138c46e7: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [WARN] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.715Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering candidate state: node="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Candidate]" term=2 testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering candidate state: node="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Candidate]" term=2 testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.716Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering candidate state: node="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Candidate]" term=2 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.716Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.716Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: duplicate requestVote for same term: term=2 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [WARN] server-5605632c-44b1-3682-0307-b526138c46e7: Election timeout reached, restarting election testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering candidate state: node="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Candidate]" term=3 testLoggerAdapter verbose: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.782Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering follower state: follower="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Follower]" leader= testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: election won: tally=2 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering leader state: leader="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Leader]" testLoggerAdapter verbose: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.782Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering follower state: follower="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Follower]" leader= testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: added peer, starting replication: peer=server-6b2c1140-0369-4a98-7c57-fbc453c8233a testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: added peer, starting replication: peer=server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: pipelining replication: peer="{Voter server-6b2c1140-0369-4a98-7c57-fbc453c8233a 6b2c1140-0369-4a98-7c57-fbc453c8233a}" testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: pipelining replication: peer="{Voter server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 fbb7ea75-9654-4c92-8f3e-87789309f4a5}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.887Z [INFO] stable state for Follower reached at 2021-10-19 21:54:23.782621629 +0000 UTC m=+4.907647755 (2 nodes), 148.406325ms from start of poll, 150.007719ms from cluster start. Timeout at 2021-10-19 21:54:23.887694909 +0000 UTC m=+5.012721051, 105.073296ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.887Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:23.992Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.887832671 +0000 UTC m=+5.012858794 (1 nodes), 617ns from start of poll, 255.218758ms from cluster start. Timeout at 2021-10-19 21:54:23.992913385 +0000 UTC m=+5.117939531, 105.080737ms after stability testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.003Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: aborting pipeline replication: peer="{Voter server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 fbb7ea75-9654-4c92-8f3e-87789309f4a5}" testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.003Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: aborting pipeline replication: peer="{Voter server-6b2c1140-0369-4a98-7c57-fbc453c8233a 6b2c1140-0369-4a98-7c57-fbc453c8233a}" testLoggerAdapter verbose: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.053Z [ERROR] server-5605632c-44b1-3682-0307-b526138c46e7: failed to heartbeat to: peer=fbb7ea75-9654-4c92-8f3e-87789309f4a5 error="command timed out" --- PASS: TestRaft_TripleNode (0.43s) testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.633Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.633Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.633Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering follower state: follower="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Follower]" leader= testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.633Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering follower state: follower="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Follower]" leader= testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.633Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6b2c1140-0369-4a98-7c57-fbc453c8233a Address:6b2c1140-0369-4a98-7c57-fbc453c8233a} {Suffrage:Voter ID:server-5605632c-44b1-3682-0307-b526138c46e7 Address:5605632c-44b1-3682-0307-b526138c46e7} {Suffrage:Voter ID:server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 Address:fbb7ea75-9654-4c92-8f3e-87789309f4a5}]" testing.go:149: cluster: 2021-10-19T21:54:23.634Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.634Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering follower state: follower="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Follower]" leader= testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.715Z [WARN] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [WARN] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.715Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering candidate state: node="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Candidate]" term=2 testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering candidate state: node="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Candidate]" term=2 testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.715Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: duplicate requestVote for same term: term=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.715Z [WARN] server-5605632c-44b1-3682-0307-b526138c46e7: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.716Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: duplicate requestVote for same term: term=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering candidate state: node="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Candidate]" term=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: duplicate requestVote for same term: term=2 testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.716Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: duplicate requestVote for same term: term=2 testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.716Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: duplicate requestVote for same term: term=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.716Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: duplicate requestVote for same term: term=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [WARN] server-5605632c-44b1-3682-0307-b526138c46e7: Election timeout reached, restarting election testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering candidate state: node="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Candidate]" term=3 testing.go:149: server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: 2021-10-19T21:54:23.782Z [INFO] server-fbb7ea75-9654-4c92-8f3e-87789309f4a5: entering follower state: follower="Node at fbb7ea75-9654-4c92-8f3e-87789309f4a5 [Follower]" leader= testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: election won: tally=2 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: entering leader state: leader="Node at 5605632c-44b1-3682-0307-b526138c46e7 [Leader]" testing.go:149: server-6b2c1140-0369-4a98-7c57-fbc453c8233a: 2021-10-19T21:54:23.782Z [INFO] server-6b2c1140-0369-4a98-7c57-fbc453c8233a: entering follower state: follower="Node at 6b2c1140-0369-4a98-7c57-fbc453c8233a [Follower]" leader= testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: added peer, starting replication: peer=server-6b2c1140-0369-4a98-7c57-fbc453c8233a testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: added peer, starting replication: peer=server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: pipelining replication: peer="{Voter server-6b2c1140-0369-4a98-7c57-fbc453c8233a 6b2c1140-0369-4a98-7c57-fbc453c8233a}" testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:23.782Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: pipelining replication: peer="{Voter server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 fbb7ea75-9654-4c92-8f3e-87789309f4a5}" testing.go:149: cluster: 2021-10-19T21:54:23.887Z [INFO] stable state for Follower reached at 2021-10-19 21:54:23.782621629 +0000 UTC m=+4.907647755 (2 nodes), 148.406325ms from start of poll, 150.007719ms from cluster start. Timeout at 2021-10-19 21:54:23.887694909 +0000 UTC m=+5.012721051, 105.073296ms after stability testing.go:149: cluster: 2021-10-19T21:54:23.887Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:23.992Z [INFO] stable state for Leader reached at 2021-10-19 21:54:23.887832671 +0000 UTC m=+5.012858794 (1 nodes), 617ns from start of poll, 255.218758ms from cluster start. Timeout at 2021-10-19 21:54:23.992913385 +0000 UTC m=+5.117939531, 105.080737ms after stability testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.003Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: aborting pipeline replication: peer="{Voter server-fbb7ea75-9654-4c92-8f3e-87789309f4a5 fbb7ea75-9654-4c92-8f3e-87789309f4a5}" testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.003Z [INFO] server-5605632c-44b1-3682-0307-b526138c46e7: aborting pipeline replication: peer="{Voter server-6b2c1140-0369-4a98-7c57-fbc453c8233a 6b2c1140-0369-4a98-7c57-fbc453c8233a}" testing.go:149: server-5605632c-44b1-3682-0307-b526138c46e7: 2021-10-19T21:54:24.053Z [ERROR] server-5605632c-44b1-3682-0307-b526138c46e7: failed to heartbeat to: peer=fbb7ea75-9654-4c92-8f3e-87789309f4a5 error="command timed out" === RUN TestRaft_LeaderFail testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.056Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.056Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.056Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.056Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering follower state: follower="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Follower]" leader= testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.057Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.057Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.057Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering follower state: follower="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Follower]" leader= testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=2 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: election won: tally=2 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering leader state: leader="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Leader]" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: added peer, starting replication: peer=server-24a99f32-862a-6641-1f39-d20b15cd7b75 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: pipelining replication: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.108Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.213Z [INFO] stable state for Follower reached at 2021-10-19 21:54:24.107810005 +0000 UTC m=+5.232836174 (2 nodes), 50.692765ms from start of poll, 51.426925ms from cluster start. Timeout at 2021-10-19 21:54:24.212897504 +0000 UTC m=+5.337923644, 105.08747ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.213Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.318Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.213121036 +0000 UTC m=+5.338147161 (1 nodes), 1.058µs from start of poll, 156.737912ms from cluster start. Timeout at 2021-10-19 21:54:24.318209929 +0000 UTC m=+5.443236101, 105.08894ms after stability testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.331Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.332Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.334Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.335Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to pipeline appendEntries: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="append pipeline closed" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.335Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: aborting pipeline replication: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.336Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to pipeline appendEntries: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="append pipeline closed" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.336Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.342Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.345Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.348Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.350Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.359Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.364Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.367Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.367Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.375Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.377Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-d9847145-32ee-9d28-17fc-f5550d4b004b time=50.079981ms testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.381Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-24a99f32-862a-6641-1f39-d20b15cd7b75 time=59.560333ms testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-d9847145-32ee-9d28-17fc-f5550d4b004b time=60.239731ms testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.394Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.401Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: heartbeat timeout reached, starting election: last-leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=3 testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.420Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: rejecting vote request since we have a leader: from=24a99f32-862a-6641-1f39-d20b15cd7b75 leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.463Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: heartbeat timeout reached, starting election: last-leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.463Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.464Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering candidate state: node="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Candidate]" term=3 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=3 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.464Z [ERROR] server-d9847145-32ee-9d28-17fc-f5550d4b004b: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.464Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: Election timeout reached, restarting election testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=4 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.506Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering follower state: follower="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Follower]" leader= testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: election won: tally=2 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering leader state: leader="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Leader]" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-54a5a1d5-0d04-fc16-000b-cec32b170021 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.514Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=4 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.525Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.531Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.543Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.550Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.557Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to contact: server-id=server-54a5a1d5-0d04-fc16-000b-cec32b170021 time=50.155523ms testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.569Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.579Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=5 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.602Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to contact: server-id=server-54a5a1d5-0d04-fc16-000b-cec32b170021 time=95.597807ms testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.612Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.507084244 +0000 UTC m=+5.632110407 (1 nodes), 172.792349ms from start of poll, 450.701158ms from cluster start. Timeout at 2021-10-19 21:54:24.612208951 +0000 UTC m=+5.737235145, 105.124738ms after stability testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.617Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: peer has newer term, stopping replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.617Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering follower state: follower="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Follower]" leader= testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.618Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.645Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.645Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=6 testLoggerAdapter verbose: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.646Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: rejecting vote request since we have a leader: from=54a5a1d5-0d04-fc16-000b-cec32b170021 leader=24a99f32-862a-6641-1f39-d20b15cd7b75 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.646Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: rejecting vote request since our last term is greater: candidate=54a5a1d5-0d04-fc16-000b-cec32b170021 last-term=4 last-candidate-term=2 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=7 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: election won: tally=2 testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.695Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering leader state: leader="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Leader]" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-54a5a1d5-0d04-fc16-000b-cec32b170021 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testLoggerAdapter verbose: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.695Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to get previous log: previous-index=5 last-index=3 error="log not found" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: appendEntries rejected, sending older logs: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" next=4 testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.701Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testLoggerAdapter verbose: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.701Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" --- PASS: TestRaft_LeaderFail (0.68s) testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.056Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.056Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.056Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.056Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering follower state: follower="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Follower]" leader= testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.057Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-54a5a1d5-0d04-fc16-000b-cec32b170021 Address:54a5a1d5-0d04-fc16-000b-cec32b170021} {Suffrage:Voter ID:server-24a99f32-862a-6641-1f39-d20b15cd7b75 Address:24a99f32-862a-6641-1f39-d20b15cd7b75} {Suffrage:Voter ID:server-d9847145-32ee-9d28-17fc-f5550d4b004b Address:d9847145-32ee-9d28-17fc-f5550d4b004b}]" testing.go:149: cluster: 2021-10-19T21:54:24.057Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.057Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering follower state: follower="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Follower]" leader= testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=2 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: election won: tally=2 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering leader state: leader="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Leader]" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: added peer, starting replication: peer=server-24a99f32-862a-6641-1f39-d20b15cd7b75 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.107Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: pipelining replication: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.108Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: cluster: 2021-10-19T21:54:24.213Z [INFO] stable state for Follower reached at 2021-10-19 21:54:24.107810005 +0000 UTC m=+5.232836174 (2 nodes), 50.692765ms from start of poll, 51.426925ms from cluster start. Timeout at 2021-10-19 21:54:24.212897504 +0000 UTC m=+5.337923644, 105.08747ms after stability testing.go:149: cluster: 2021-10-19T21:54:24.213Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:24.318Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.213121036 +0000 UTC m=+5.338147161 (1 nodes), 1.058µs from start of poll, 156.737912ms from cluster start. Timeout at 2021-10-19 21:54:24.318209929 +0000 UTC m=+5.443236101, 105.08894ms after stability raft_test.go:336: [INFO] Disconnecting Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Leader] testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.331Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.332Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: cluster: 2021-10-19T21:54:24.334Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.335Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to pipeline appendEntries: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="append pipeline closed" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.335Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: aborting pipeline replication: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.336Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to pipeline appendEntries: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="append pipeline closed" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.336Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.342Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.345Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.348Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.350Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.359Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.364Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.367Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=d9847145-32ee-9d28-17fc-f5550d4b004b error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.367Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.375Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.377Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-d9847145-32ee-9d28-17fc-f5550d4b004b time=50.079981ms testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.381Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-24a99f32-862a-6641-1f39-d20b15cd7b75 time=59.560333ms testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact: server-id=server-d9847145-32ee-9d28-17fc-f5550d4b004b time=60.239731ms testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to contact quorum of nodes, stepping down testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.387Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.394Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to heartbeat to: peer=24a99f32-862a-6641-1f39-d20b15cd7b75 error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.401Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to appendEntries to: peer="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: heartbeat timeout reached, starting election: last-leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=3 testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.420Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: rejecting vote request since we have a leader: from=24a99f32-862a-6641-1f39-d20b15cd7b75 leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.420Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.463Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: heartbeat timeout reached, starting election: last-leader=54a5a1d5-0d04-fc16-000b-cec32b170021 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.463Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=3 testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.464Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering candidate state: node="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Candidate]" term=3 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.464Z [ERROR] server-d9847145-32ee-9d28-17fc-f5550d4b004b: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.464Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.464Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: duplicate requestVote for same term: term=3 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: Election timeout reached, restarting election testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=4 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to make requestVote RPC: target="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.506Z [INFO] server-d9847145-32ee-9d28-17fc-f5550d4b004b: entering follower state: follower="Node at d9847145-32ee-9d28-17fc-f5550d4b004b [Follower]" leader= testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: election won: tally=2 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.506Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering leader state: leader="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Leader]" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-54a5a1d5-0d04-fc16-000b-cec32b170021 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.507Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.514Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=4 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.522Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.525Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.531Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.543Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.550Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.557Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to contact: server-id=server-54a5a1d5-0d04-fc16-000b-cec32b170021 time=50.155523ms testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.569Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to appendEntries to: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.579Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to heartbeat to: peer=54a5a1d5-0d04-fc16-000b-cec32b170021 error="failed to connect to peer: 54a5a1d5-0d04-fc16-000b-cec32b170021" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=5 testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" error="failed to connect to peer: d9847145-32ee-9d28-17fc-f5550d4b004b" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.587Z [ERROR] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to make requestVote RPC: target="{Voter server-24a99f32-862a-6641-1f39-d20b15cd7b75 24a99f32-862a-6641-1f39-d20b15cd7b75}" error="failed to connect to peer: 24a99f32-862a-6641-1f39-d20b15cd7b75" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.602Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: failed to contact: server-id=server-54a5a1d5-0d04-fc16-000b-cec32b170021 time=95.597807ms testing.go:149: cluster: 2021-10-19T21:54:24.612Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.507084244 +0000 UTC m=+5.632110407 (1 nodes), 172.792349ms from start of poll, 450.701158ms from cluster start. Timeout at 2021-10-19 21:54:24.612208951 +0000 UTC m=+5.737235145, 105.124738ms after stability raft_test.go:371: [INFO] Reconnecting Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate] testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.617Z [ERROR] server-24a99f32-862a-6641-1f39-d20b15cd7b75: peer has newer term, stopping replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.617Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering follower state: follower="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Follower]" leader= testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.618Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.645Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: Election timeout reached, restarting election testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.645Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering candidate state: node="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Candidate]" term=6 testing.go:149: server-d9847145-32ee-9d28-17fc-f5550d4b004b: 2021-10-19T21:54:24.646Z [WARN] server-d9847145-32ee-9d28-17fc-f5550d4b004b: rejecting vote request since we have a leader: from=54a5a1d5-0d04-fc16-000b-cec32b170021 leader=24a99f32-862a-6641-1f39-d20b15cd7b75 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.646Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: rejecting vote request since our last term is greater: candidate=54a5a1d5-0d04-fc16-000b-cec32b170021 last-term=4 last-candidate-term=2 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering candidate state: node="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Candidate]" term=7 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: election won: tally=2 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: entering leader state: leader="Node at 24a99f32-862a-6641-1f39-d20b15cd7b75 [Leader]" testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.695Z [INFO] server-54a5a1d5-0d04-fc16-000b-cec32b170021: entering follower state: follower="Node at 54a5a1d5-0d04-fc16-000b-cec32b170021 [Follower]" leader= testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-54a5a1d5-0d04-fc16-000b-cec32b170021 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.695Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: added peer, starting replication: peer=server-d9847145-32ee-9d28-17fc-f5550d4b004b testing.go:149: server-54a5a1d5-0d04-fc16-000b-cec32b170021: 2021-10-19T21:54:24.695Z [WARN] server-54a5a1d5-0d04-fc16-000b-cec32b170021: failed to get previous log: previous-index=5 last-index=3 error="log not found" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [WARN] server-24a99f32-862a-6641-1f39-d20b15cd7b75: appendEntries rejected, sending older logs: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" next=4 testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.696Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: pipelining replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.701Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-d9847145-32ee-9d28-17fc-f5550d4b004b d9847145-32ee-9d28-17fc-f5550d4b004b}" testing.go:149: server-24a99f32-862a-6641-1f39-d20b15cd7b75: 2021-10-19T21:54:24.701Z [INFO] server-24a99f32-862a-6641-1f39-d20b15cd7b75: aborting pipeline replication: peer="{Voter server-54a5a1d5-0d04-fc16-000b-cec32b170021 54a5a1d5-0d04-fc16-000b-cec32b170021}" === RUN TestRaft_BehindFollower testLoggerAdapter verbose: server-7d00f43f-d0de-1684-c58c-a354983465b5: 2021-10-19T21:54:24.732Z [INFO] server-7d00f43f-d0de-1684-c58c-a354983465b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testLoggerAdapter verbose: server-7d00f43f-d0de-1684-c58c-a354983465b5: 2021-10-19T21:54:24.732Z [INFO] server-7d00f43f-d0de-1684-c58c-a354983465b5: entering follower state: follower="Node at 7d00f43f-d0de-1684-c58c-a354983465b5 [Follower]" leader= testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.733Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.733Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering follower state: follower="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Follower]" leader= testLoggerAdapter verbose: server-41653917-269e-9f7e-d588-cb535cc8fe3a: 2021-10-19T21:54:24.733Z [INFO] server-41653917-269e-9f7e-d588-cb535cc8fe3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.733Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-41653917-269e-9f7e-d588-cb535cc8fe3a: 2021-10-19T21:54:24.733Z [INFO] server-41653917-269e-9f7e-d588-cb535cc8fe3a: entering follower state: follower="Node at 41653917-269e-9f7e-d588-cb535cc8fe3a [Follower]" leader= testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.810Z [WARN] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.810Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering candidate state: node="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Candidate]" term=2 testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: election won: tally=2 testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering leader state: leader="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Leader]" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: added peer, starting replication: peer=server-7d00f43f-d0de-1684-c58c-a354983465b5 testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: added peer, starting replication: peer=server-41653917-269e-9f7e-d588-cb535cc8fe3a testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-41653917-269e-9f7e-d588-cb535cc8fe3a 41653917-269e-9f7e-d588-cb535cc8fe3a}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.916Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.811372318 +0000 UTC m=+5.936398509 (1 nodes), 78.022288ms from start of poll, 78.870745ms from cluster start. Timeout at 2021-10-19 21:54:24.916481408 +0000 UTC m=+6.041507603, 105.109094ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:24.916Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.022Z [INFO] stable state for Follower reached at 2021-10-19 21:54:24.916817227 +0000 UTC m=+6.041843379 (2 nodes), 1.94µs from start of poll, 184.315615ms from cluster start. Timeout at 2021-10-19 21:54:25.021904494 +0000 UTC m=+6.146930705, 105.087326ms after stability testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to pipeline appendEntries: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" error="append pipeline closed" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to appendEntries to: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" error="failed to connect to peer: 7d00f43f-d0de-1684-c58c-a354983465b5" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.026Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to heartbeat to: peer=7d00f43f-d0de-1684-c58c-a354983465b5 error="failed to connect to peer: 7d00f43f-d0de-1684-c58c-a354983465b5" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.034Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.042Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.149Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.043110113 +0000 UTC m=+6.168136241 (1 nodes), 1.786µs from start of poll, 310.608477ms from cluster start. Timeout at 2021-10-19 21:54:25.149468084 +0000 UTC m=+6.274494261, 106.35802ms after stability testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.149Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-41653917-269e-9f7e-d588-cb535cc8fe3a 41653917-269e-9f7e-d588-cb535cc8fe3a}" testLoggerAdapter verbose: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.149Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" --- PASS: TestRaft_BehindFollower (0.42s) testing.go:149: server-7d00f43f-d0de-1684-c58c-a354983465b5: 2021-10-19T21:54:24.732Z [INFO] server-7d00f43f-d0de-1684-c58c-a354983465b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testing.go:149: server-7d00f43f-d0de-1684-c58c-a354983465b5: 2021-10-19T21:54:24.732Z [INFO] server-7d00f43f-d0de-1684-c58c-a354983465b5: entering follower state: follower="Node at 7d00f43f-d0de-1684-c58c-a354983465b5 [Follower]" leader= testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.733Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.733Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering follower state: follower="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Follower]" leader= testing.go:149: server-41653917-269e-9f7e-d588-cb535cc8fe3a: 2021-10-19T21:54:24.733Z [INFO] server-41653917-269e-9f7e-d588-cb535cc8fe3a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7d00f43f-d0de-1684-c58c-a354983465b5 Address:7d00f43f-d0de-1684-c58c-a354983465b5} {Suffrage:Voter ID:server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51 Address:72f2dab5-e0b8-7ce4-7f81-1c041c588d51} {Suffrage:Voter ID:server-41653917-269e-9f7e-d588-cb535cc8fe3a Address:41653917-269e-9f7e-d588-cb535cc8fe3a}]" testing.go:149: cluster: 2021-10-19T21:54:24.733Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-41653917-269e-9f7e-d588-cb535cc8fe3a: 2021-10-19T21:54:24.733Z [INFO] server-41653917-269e-9f7e-d588-cb535cc8fe3a: entering follower state: follower="Node at 41653917-269e-9f7e-d588-cb535cc8fe3a [Follower]" leader= testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.810Z [WARN] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.810Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering candidate state: node="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Candidate]" term=2 testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: election won: tally=2 testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: entering leader state: leader="Node at 72f2dab5-e0b8-7ce4-7f81-1c041c588d51 [Leader]" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: added peer, starting replication: peer=server-7d00f43f-d0de-1684-c58c-a354983465b5 testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: added peer, starting replication: peer=server-41653917-269e-9f7e-d588-cb535cc8fe3a testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:24.811Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-41653917-269e-9f7e-d588-cb535cc8fe3a 41653917-269e-9f7e-d588-cb535cc8fe3a}" testing.go:149: cluster: 2021-10-19T21:54:24.916Z [INFO] stable state for Leader reached at 2021-10-19 21:54:24.811372318 +0000 UTC m=+5.936398509 (1 nodes), 78.022288ms from start of poll, 78.870745ms from cluster start. Timeout at 2021-10-19 21:54:24.916481408 +0000 UTC m=+6.041507603, 105.109094ms after stability testing.go:149: cluster: 2021-10-19T21:54:24.916Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:25.022Z [INFO] stable state for Follower reached at 2021-10-19 21:54:24.916817227 +0000 UTC m=+6.041843379 (2 nodes), 1.94µs from start of poll, 184.315615ms from cluster start. Timeout at 2021-10-19 21:54:25.021904494 +0000 UTC m=+6.146930705, 105.087326ms after stability testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to pipeline appendEntries: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" error="append pipeline closed" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.022Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to appendEntries to: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" error="failed to connect to peer: 7d00f43f-d0de-1684-c58c-a354983465b5" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.026Z [ERROR] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: failed to heartbeat to: peer=7d00f43f-d0de-1684-c58c-a354983465b5 error="failed to connect to peer: 7d00f43f-d0de-1684-c58c-a354983465b5" raft_test.go:420: [INFO] Finished apply without behind follower testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.034Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: pipelining replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" testing.go:149: cluster: 2021-10-19T21:54:25.042Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:25.149Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.043110113 +0000 UTC m=+6.168136241 (1 nodes), 1.786µs from start of poll, 310.608477ms from cluster start. Timeout at 2021-10-19 21:54:25.149468084 +0000 UTC m=+6.274494261, 106.35802ms after stability testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.149Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-41653917-269e-9f7e-d588-cb535cc8fe3a 41653917-269e-9f7e-d588-cb535cc8fe3a}" testing.go:149: server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: 2021-10-19T21:54:25.149Z [INFO] server-72f2dab5-e0b8-7ce4-7f81-1c041c588d51: aborting pipeline replication: peer="{Voter server-7d00f43f-d0de-1684-c58c-a354983465b5 7d00f43f-d0de-1684-c58c-a354983465b5}" === RUN TestRaft_ApplyNonLeader testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.152Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.152Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering follower state: follower="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Follower]" leader= testLoggerAdapter verbose: server-ed5750d3-3814-65d4-9b69-f809bfc63179: 2021-10-19T21:54:25.152Z [INFO] server-ed5750d3-3814-65d4-9b69-f809bfc63179: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testLoggerAdapter verbose: server-ed5750d3-3814-65d4-9b69-f809bfc63179: 2021-10-19T21:54:25.153Z [INFO] server-ed5750d3-3814-65d4-9b69-f809bfc63179: entering follower state: follower="Node at ed5750d3-3814-65d4-9b69-f809bfc63179 [Follower]" leader= testLoggerAdapter verbose: server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: 2021-10-19T21:54:25.153Z [INFO] server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.153Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: 2021-10-19T21:54:25.153Z [INFO] server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: entering follower state: follower="Node at c3f587bc-563c-66e3-47d6-8fb70c1accb1 [Follower]" leader= testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.212Z [WARN] server-44328be5-6f57-cb63-e4e6-708a543ccfec: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.212Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering candidate state: node="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Candidate]" term=2 testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: election won: tally=2 testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering leader state: leader="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Leader]" testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: added peer, starting replication: peer=server-ed5750d3-3814-65d4-9b69-f809bfc63179 testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: added peer, starting replication: peer=server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: pipelining replication: peer="{Voter server-ed5750d3-3814-65d4-9b69-f809bfc63179 ed5750d3-3814-65d4-9b69-f809bfc63179}" testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: pipelining replication: peer="{Voter server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 c3f587bc-563c-66e3-47d6-8fb70c1accb1}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.318Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.213222052 +0000 UTC m=+6.338248187 (1 nodes), 60.003248ms from start of poll, 60.698391ms from cluster start. Timeout at 2021-10-19 21:54:25.31836261 +0000 UTC m=+6.443388801, 105.140614ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.318Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.423Z [INFO] stable state for Follower reached at 2021-10-19 21:54:25.318665729 +0000 UTC m=+6.443691856 (2 nodes), 2.479µs from start of poll, 166.14206ms from cluster start. Timeout at 2021-10-19 21:54:25.423749077 +0000 UTC m=+6.548775235, 105.083379ms after stability testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.424Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: aborting pipeline replication: peer="{Voter server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 c3f587bc-563c-66e3-47d6-8fb70c1accb1}" testLoggerAdapter verbose: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.424Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: aborting pipeline replication: peer="{Voter server-ed5750d3-3814-65d4-9b69-f809bfc63179 ed5750d3-3814-65d4-9b69-f809bfc63179}" --- PASS: TestRaft_ApplyNonLeader (0.27s) testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.152Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.152Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering follower state: follower="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Follower]" leader= testing.go:149: server-ed5750d3-3814-65d4-9b69-f809bfc63179: 2021-10-19T21:54:25.152Z [INFO] server-ed5750d3-3814-65d4-9b69-f809bfc63179: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testing.go:149: server-ed5750d3-3814-65d4-9b69-f809bfc63179: 2021-10-19T21:54:25.153Z [INFO] server-ed5750d3-3814-65d4-9b69-f809bfc63179: entering follower state: follower="Node at ed5750d3-3814-65d4-9b69-f809bfc63179 [Follower]" leader= testing.go:149: server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: 2021-10-19T21:54:25.153Z [INFO] server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-44328be5-6f57-cb63-e4e6-708a543ccfec Address:44328be5-6f57-cb63-e4e6-708a543ccfec} {Suffrage:Voter ID:server-ed5750d3-3814-65d4-9b69-f809bfc63179 Address:ed5750d3-3814-65d4-9b69-f809bfc63179} {Suffrage:Voter ID:server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 Address:c3f587bc-563c-66e3-47d6-8fb70c1accb1}]" testing.go:149: cluster: 2021-10-19T21:54:25.153Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: 2021-10-19T21:54:25.153Z [INFO] server-c3f587bc-563c-66e3-47d6-8fb70c1accb1: entering follower state: follower="Node at c3f587bc-563c-66e3-47d6-8fb70c1accb1 [Follower]" leader= testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.212Z [WARN] server-44328be5-6f57-cb63-e4e6-708a543ccfec: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.212Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering candidate state: node="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Candidate]" term=2 testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: election won: tally=2 testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: entering leader state: leader="Node at 44328be5-6f57-cb63-e4e6-708a543ccfec [Leader]" testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: added peer, starting replication: peer=server-ed5750d3-3814-65d4-9b69-f809bfc63179 testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: added peer, starting replication: peer=server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: pipelining replication: peer="{Voter server-ed5750d3-3814-65d4-9b69-f809bfc63179 ed5750d3-3814-65d4-9b69-f809bfc63179}" testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.213Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: pipelining replication: peer="{Voter server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 c3f587bc-563c-66e3-47d6-8fb70c1accb1}" testing.go:149: cluster: 2021-10-19T21:54:25.318Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.213222052 +0000 UTC m=+6.338248187 (1 nodes), 60.003248ms from start of poll, 60.698391ms from cluster start. Timeout at 2021-10-19 21:54:25.31836261 +0000 UTC m=+6.443388801, 105.140614ms after stability testing.go:149: cluster: 2021-10-19T21:54:25.318Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:25.423Z [INFO] stable state for Follower reached at 2021-10-19 21:54:25.318665729 +0000 UTC m=+6.443691856 (2 nodes), 2.479µs from start of poll, 166.14206ms from cluster start. Timeout at 2021-10-19 21:54:25.423749077 +0000 UTC m=+6.548775235, 105.083379ms after stability testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.424Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: aborting pipeline replication: peer="{Voter server-c3f587bc-563c-66e3-47d6-8fb70c1accb1 c3f587bc-563c-66e3-47d6-8fb70c1accb1}" testing.go:149: server-44328be5-6f57-cb63-e4e6-708a543ccfec: 2021-10-19T21:54:25.424Z [INFO] server-44328be5-6f57-cb63-e4e6-708a543ccfec: aborting pipeline replication: peer="{Voter server-ed5750d3-3814-65d4-9b69-f809bfc63179 ed5750d3-3814-65d4-9b69-f809bfc63179}" === RUN TestRaft_ApplyConcurrent testLoggerAdapter verbose: server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: 2021-10-19T21:54:25.428Z [INFO] server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testLoggerAdapter verbose: server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: 2021-10-19T21:54:25.428Z [INFO] server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: entering follower state: follower="Node at 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f [Follower]" leader= testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.428Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.428Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering follower state: follower="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Follower]" leader= testLoggerAdapter verbose: server-ee446150-82e7-71ae-f074-14094619f8d9: 2021-10-19T21:54:25.428Z [INFO] server-ee446150-82e7-71ae-f074-14094619f8d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.428Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-ee446150-82e7-71ae-f074-14094619f8d9: 2021-10-19T21:54:25.428Z [INFO] server-ee446150-82e7-71ae-f074-14094619f8d9: entering follower state: follower="Node at ee446150-82e7-71ae-f074-14094619f8d9 [Follower]" leader= testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [WARN] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering candidate state: node="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Candidate]" term=2 testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: election won: tally=2 testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering leader state: leader="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Leader]" testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: added peer, starting replication: peer=server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: added peer, starting replication: peer=server-ee446150-82e7-71ae-f074-14094619f8d9 testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: pipelining replication: peer="{Voter server-ee446150-82e7-71ae-f074-14094619f8d9 ee446150-82e7-71ae-f074-14094619f8d9}" testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.541Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: pipelining replication: peer="{Voter server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.745Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.540704388 +0000 UTC m=+6.665730592 (1 nodes), 111.825389ms from start of poll, 112.815598ms from cluster start. Timeout at 2021-10-19 21:54:25.745773272 +0000 UTC m=+6.870799417, 205.068825ms after stability testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.758Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: aborting pipeline replication: peer="{Voter server-ee446150-82e7-71ae-f074-14094619f8d9 ee446150-82e7-71ae-f074-14094619f8d9}" testLoggerAdapter verbose: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.758Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: aborting pipeline replication: peer="{Voter server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f}" --- PASS: TestRaft_ApplyConcurrent (0.33s) testing.go:149: server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: 2021-10-19T21:54:25.428Z [INFO] server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testing.go:149: server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: 2021-10-19T21:54:25.428Z [INFO] server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f: entering follower state: follower="Node at 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f [Follower]" leader= testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.428Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.428Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering follower state: follower="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Follower]" leader= testing.go:149: server-ee446150-82e7-71ae-f074-14094619f8d9: 2021-10-19T21:54:25.428Z [INFO] server-ee446150-82e7-71ae-f074-14094619f8d9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f Address:50d10a7d-13d9-f919-1ccf-3d8ca0ef369f} {Suffrage:Voter ID:server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 Address:8f02f141-c72a-5d79-c04e-1bfd3b5d96d8} {Suffrage:Voter ID:server-ee446150-82e7-71ae-f074-14094619f8d9 Address:ee446150-82e7-71ae-f074-14094619f8d9}]" testing.go:149: cluster: 2021-10-19T21:54:25.428Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-ee446150-82e7-71ae-f074-14094619f8d9: 2021-10-19T21:54:25.428Z [INFO] server-ee446150-82e7-71ae-f074-14094619f8d9: entering follower state: follower="Node at ee446150-82e7-71ae-f074-14094619f8d9 [Follower]" leader= testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [WARN] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering candidate state: node="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Candidate]" term=2 testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: election won: tally=2 testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: entering leader state: leader="Node at 8f02f141-c72a-5d79-c04e-1bfd3b5d96d8 [Leader]" testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: added peer, starting replication: peer=server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: added peer, starting replication: peer=server-ee446150-82e7-71ae-f074-14094619f8d9 testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.540Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: pipelining replication: peer="{Voter server-ee446150-82e7-71ae-f074-14094619f8d9 ee446150-82e7-71ae-f074-14094619f8d9}" testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.541Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: pipelining replication: peer="{Voter server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f}" testing.go:149: cluster: 2021-10-19T21:54:25.745Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.540704388 +0000 UTC m=+6.665730592 (1 nodes), 111.825389ms from start of poll, 112.815598ms from cluster start. Timeout at 2021-10-19 21:54:25.745773272 +0000 UTC m=+6.870799417, 205.068825ms after stability testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.758Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: aborting pipeline replication: peer="{Voter server-ee446150-82e7-71ae-f074-14094619f8d9 ee446150-82e7-71ae-f074-14094619f8d9}" testing.go:149: server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: 2021-10-19T21:54:25.758Z [INFO] server-8f02f141-c72a-5d79-c04e-1bfd3b5d96d8: aborting pipeline replication: peer="{Voter server-50d10a7d-13d9-f919-1ccf-3d8ca0ef369f 50d10a7d-13d9-f919-1ccf-3d8ca0ef369f}" === RUN TestRaft_ApplyConcurrent_Timeout testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.760Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ebcde72-d136-a8b1-fe35-fe41d42be50a Address:2ebcde72-d136-a8b1-fe35-fe41d42be50a}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:25.760Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.760Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering follower state: follower="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Follower]" leader= testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [WARN] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering candidate state: node="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Candidate]" term=2 testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: election won: tally=1 testLoggerAdapter verbose: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering leader state: leader="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.155Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.954392149 +0000 UTC m=+7.079418277 (1 nodes), 194.071349ms from start of poll, 194.346787ms from cluster start. Timeout at 2021-10-19 21:54:26.155486239 +0000 UTC m=+7.280512379, 201.094102ms after stability --- PASS: TestRaft_ApplyConcurrent_Timeout (0.40s) testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.760Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2ebcde72-d136-a8b1-fe35-fe41d42be50a Address:2ebcde72-d136-a8b1-fe35-fe41d42be50a}]" testing.go:149: cluster: 2021-10-19T21:54:25.760Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.760Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering follower state: follower="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Follower]" leader= testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [WARN] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering candidate state: node="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Candidate]" term=2 testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: election won: tally=1 testing.go:149: server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: 2021-10-19T21:54:25.954Z [INFO] server-2ebcde72-d136-a8b1-fe35-fe41d42be50a: entering leader state: leader="Node at 2ebcde72-d136-a8b1-fe35-fe41d42be50a [Leader]" testing.go:149: cluster: 2021-10-19T21:54:26.155Z [INFO] stable state for Leader reached at 2021-10-19 21:54:25.954392149 +0000 UTC m=+7.079418277 (1 nodes), 194.071349ms from start of poll, 194.346787ms from cluster start. Timeout at 2021-10-19 21:54:26.155486239 +0000 UTC m=+7.280512379, 201.094102ms after stability === RUN TestRaft_JoinNode testLoggerAdapter verbose: server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: 2021-10-19T21:54:26.157Z [INFO] server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25}]" testLoggerAdapter verbose: server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: 2021-10-19T21:54:26.157Z [INFO] server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: entering follower state: follower="Node at 7603a38c-4b42-4453-03f3-a2a9877e2d7a [Follower]" leader= testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.157Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25}]" testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.158Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering follower state: follower="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Follower]" leader= testLoggerAdapter verbose: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.158Z [INFO] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.158Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.158Z [INFO] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: entering follower state: follower="Node at b7b521c3-9659-b83e-6d66-c62fe4e347c5 [Follower]" leader= testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.225Z [WARN] server-350a5799-753c-7c46-9a03-cadfb0b33a25: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering candidate state: node="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Candidate]" term=2 testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: election won: tally=2 testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering leader state: leader="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Leader]" testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: added peer, starting replication: peer=server-7603a38c-4b42-4453-03f3-a2a9877e2d7a testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: pipelining replication: peer="{Voter server-7603a38c-4b42-4453-03f3-a2a9877e2d7a 7603a38c-4b42-4453-03f3-a2a9877e2d7a}" testLoggerAdapter verbose: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.249Z [WARN] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.331Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.226426577 +0000 UTC m=+7.351452720 (1 nodes), 67.448499ms from start of poll, 68.858253ms from cluster start. Timeout at 2021-10-19 21:54:26.331515705 +0000 UTC m=+7.456541853, 105.089133ms after stability testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.331Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: updating configuration: command=AddStaging server-id=server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 server-addr=b7b521c3-9659-b83e-6d66-c62fe4e347c5 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25} {Suffrage:Voter ID:server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 Address:b7b521c3-9659-b83e-6d66-c62fe4e347c5}]" testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.331Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: added peer, starting replication: peer=server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 testLoggerAdapter verbose: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.332Z [WARN] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.332Z [WARN] server-350a5799-753c-7c46-9a03-cadfb0b33a25: appendEntries rejected, sending older logs: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" next=1 testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.332Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: pipelining replication: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.332Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.437Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.332583862 +0000 UTC m=+7.457609987 (1 nodes), 1.231µs from start of poll, 175.01552ms from cluster start. Timeout at 2021-10-19 21:54:26.437596395 +0000 UTC m=+7.562622518, 105.012531ms after stability testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.437Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: aborting pipeline replication: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" testLoggerAdapter verbose: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.437Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: aborting pipeline replication: peer="{Voter server-7603a38c-4b42-4453-03f3-a2a9877e2d7a 7603a38c-4b42-4453-03f3-a2a9877e2d7a}" --- PASS: TestRaft_JoinNode (0.28s) testing.go:149: server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: 2021-10-19T21:54:26.157Z [INFO] server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25}]" testing.go:149: server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: 2021-10-19T21:54:26.157Z [INFO] server-7603a38c-4b42-4453-03f3-a2a9877e2d7a: entering follower state: follower="Node at 7603a38c-4b42-4453-03f3-a2a9877e2d7a [Follower]" leader= testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.157Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25}]" testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.158Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering follower state: follower="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Follower]" leader= testing.go:149: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.158Z [INFO] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-19T21:54:26.158Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.158Z [INFO] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: entering follower state: follower="Node at b7b521c3-9659-b83e-6d66-c62fe4e347c5 [Follower]" leader= testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.225Z [WARN] server-350a5799-753c-7c46-9a03-cadfb0b33a25: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering candidate state: node="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Candidate]" term=2 testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: election won: tally=2 testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: entering leader state: leader="Node at 350a5799-753c-7c46-9a03-cadfb0b33a25 [Leader]" testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: added peer, starting replication: peer=server-7603a38c-4b42-4453-03f3-a2a9877e2d7a testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.226Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: pipelining replication: peer="{Voter server-7603a38c-4b42-4453-03f3-a2a9877e2d7a 7603a38c-4b42-4453-03f3-a2a9877e2d7a}" testing.go:149: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.249Z [WARN] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: no known peers, aborting election testing.go:149: cluster: 2021-10-19T21:54:26.331Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.226426577 +0000 UTC m=+7.351452720 (1 nodes), 67.448499ms from start of poll, 68.858253ms from cluster start. Timeout at 2021-10-19 21:54:26.331515705 +0000 UTC m=+7.456541853, 105.089133ms after stability testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.331Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: updating configuration: command=AddStaging server-id=server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 server-addr=b7b521c3-9659-b83e-6d66-c62fe4e347c5 servers="[{Suffrage:Voter ID:server-7603a38c-4b42-4453-03f3-a2a9877e2d7a Address:7603a38c-4b42-4453-03f3-a2a9877e2d7a} {Suffrage:Voter ID:server-350a5799-753c-7c46-9a03-cadfb0b33a25 Address:350a5799-753c-7c46-9a03-cadfb0b33a25} {Suffrage:Voter ID:server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 Address:b7b521c3-9659-b83e-6d66-c62fe4e347c5}]" testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.331Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: added peer, starting replication: peer=server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 testing.go:149: server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: 2021-10-19T21:54:26.332Z [WARN] server-b7b521c3-9659-b83e-6d66-c62fe4e347c5: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.332Z [WARN] server-350a5799-753c-7c46-9a03-cadfb0b33a25: appendEntries rejected, sending older logs: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" next=1 testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.332Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: pipelining replication: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" testing.go:149: cluster: 2021-10-19T21:54:26.332Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:26.437Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.332583862 +0000 UTC m=+7.457609987 (1 nodes), 1.231µs from start of poll, 175.01552ms from cluster start. Timeout at 2021-10-19 21:54:26.437596395 +0000 UTC m=+7.562622518, 105.012531ms after stability testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.437Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: aborting pipeline replication: peer="{Voter server-b7b521c3-9659-b83e-6d66-c62fe4e347c5 b7b521c3-9659-b83e-6d66-c62fe4e347c5}" testing.go:149: server-350a5799-753c-7c46-9a03-cadfb0b33a25: 2021-10-19T21:54:26.437Z [INFO] server-350a5799-753c-7c46-9a03-cadfb0b33a25: aborting pipeline replication: peer="{Voter server-7603a38c-4b42-4453-03f3-a2a9877e2d7a 7603a38c-4b42-4453-03f3-a2a9877e2d7a}" === RUN TestRaft_JoinNode_ConfigStore testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.439Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485}]" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.440Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering follower state: follower="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Follower]" leader= testLoggerAdapter verbose: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.440Z [INFO] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: initial configuration: index=0 servers=[] testLoggerAdapter verbose: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.440Z [INFO] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: entering follower state: follower="Node at 267edc5a-eeec-9afa-3668-0d56dca9d72d [Follower]" leader= testLoggerAdapter verbose: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.441Z [INFO] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.441Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.441Z [INFO] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: entering follower state: follower="Node at 4c3ee407-43b8-039b-240c-1f376cf7c6d3 [Follower]" leader= testLoggerAdapter verbose: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.493Z [WARN] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: no known peers, aborting election testLoggerAdapter verbose: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.507Z [WARN] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: no known peers, aborting election testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering candidate state: node="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Candidate]" term=2 testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: election won: tally=1 testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering leader state: leader="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.625Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.519937356 +0000 UTC m=+7.644963524 (1 nodes), 78.647705ms from start of poll, 80.37562ms from cluster start. Timeout at 2021-10-19 21:54:26.625058325 +0000 UTC m=+7.750084480, 105.120956ms after stability testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: updating configuration: command=AddStaging server-id=server-267edc5a-eeec-9afa-3668-0d56dca9d72d server-addr=267edc5a-eeec-9afa-3668-0d56dca9d72d servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485} {Suffrage:Voter ID:server-267edc5a-eeec-9afa-3668-0d56dca9d72d Address:267edc5a-eeec-9afa-3668-0d56dca9d72d}]" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: added peer, starting replication: peer=server-267edc5a-eeec-9afa-3668-0d56dca9d72d testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.625Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.625Z [WARN] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: appendEntries rejected, sending older logs: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" next=1 testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: pipelining replication: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.731Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.625716874 +0000 UTC m=+7.750742999 (1 nodes), 1.046µs from start of poll, 186.155095ms from cluster start. Timeout at 2021-10-19 21:54:26.731580085 +0000 UTC m=+7.856606250, 105.863251ms after stability testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.731Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: updating configuration: command=AddStaging server-id=server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 server-addr=4c3ee407-43b8-039b-240c-1f376cf7c6d3 servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485} {Suffrage:Voter ID:server-267edc5a-eeec-9afa-3668-0d56dca9d72d Address:267edc5a-eeec-9afa-3668-0d56dca9d72d} {Suffrage:Voter ID:server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 Address:4c3ee407-43b8-039b-240c-1f376cf7c6d3}]" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: added peer, starting replication: peer=server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 testLoggerAdapter verbose: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.732Z [WARN] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: failed to get previous log: previous-index=4 last-index=0 error="log not found" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: appendEntries rejected, sending older logs: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" next=1 testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.732Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: pipelining replication: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.837Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.732832098 +0000 UTC m=+7.857858259 (1 nodes), 1.542µs from start of poll, 293.270355ms from cluster start. Timeout at 2021-10-19 21:54:26.837921864 +0000 UTC m=+7.962948011, 105.089752ms after stability testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.838Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: aborting pipeline replication: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" testLoggerAdapter verbose: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.838Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: aborting pipeline replication: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" --- PASS: TestRaft_JoinNode_ConfigStore (0.40s) testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.439Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485}]" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.440Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering follower state: follower="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Follower]" leader= testing.go:149: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.440Z [INFO] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: initial configuration: index=0 servers=[] testing.go:149: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.440Z [INFO] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: entering follower state: follower="Node at 267edc5a-eeec-9afa-3668-0d56dca9d72d [Follower]" leader= testing.go:149: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.441Z [INFO] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-19T21:54:26.441Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.441Z [INFO] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: entering follower state: follower="Node at 4c3ee407-43b8-039b-240c-1f376cf7c6d3 [Follower]" leader= testing.go:149: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.493Z [WARN] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: no known peers, aborting election testing.go:149: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.507Z [WARN] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: no known peers, aborting election testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering candidate state: node="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Candidate]" term=2 testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: election won: tally=1 testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.519Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: entering leader state: leader="Node at dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 [Leader]" testing.go:149: cluster: 2021-10-19T21:54:26.625Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.519937356 +0000 UTC m=+7.644963524 (1 nodes), 78.647705ms from start of poll, 80.37562ms from cluster start. Timeout at 2021-10-19 21:54:26.625058325 +0000 UTC m=+7.750084480, 105.120956ms after stability testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: updating configuration: command=AddStaging server-id=server-267edc5a-eeec-9afa-3668-0d56dca9d72d server-addr=267edc5a-eeec-9afa-3668-0d56dca9d72d servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485} {Suffrage:Voter ID:server-267edc5a-eeec-9afa-3668-0d56dca9d72d Address:267edc5a-eeec-9afa-3668-0d56dca9d72d}]" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: added peer, starting replication: peer=server-267edc5a-eeec-9afa-3668-0d56dca9d72d testing.go:149: cluster: 2021-10-19T21:54:26.625Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-267edc5a-eeec-9afa-3668-0d56dca9d72d: 2021-10-19T21:54:26.625Z [WARN] server-267edc5a-eeec-9afa-3668-0d56dca9d72d: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: appendEntries rejected, sending older logs: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" next=1 testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.625Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: pipelining replication: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" testing.go:149: cluster: 2021-10-19T21:54:26.731Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.625716874 +0000 UTC m=+7.750742999 (1 nodes), 1.046µs from start of poll, 186.155095ms from cluster start. Timeout at 2021-10-19 21:54:26.731580085 +0000 UTC m=+7.856606250, 105.863251ms after stability testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.731Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: updating configuration: command=AddStaging server-id=server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 server-addr=4c3ee407-43b8-039b-240c-1f376cf7c6d3 servers="[{Suffrage:Voter ID:server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485 Address:dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485} {Suffrage:Voter ID:server-267edc5a-eeec-9afa-3668-0d56dca9d72d Address:267edc5a-eeec-9afa-3668-0d56dca9d72d} {Suffrage:Voter ID:server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 Address:4c3ee407-43b8-039b-240c-1f376cf7c6d3}]" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: added peer, starting replication: peer=server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 testing.go:149: server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: 2021-10-19T21:54:26.732Z [WARN] server-4c3ee407-43b8-039b-240c-1f376cf7c6d3: failed to get previous log: previous-index=4 last-index=0 error="log not found" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [WARN] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: appendEntries rejected, sending older logs: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" next=1 testing.go:149: cluster: 2021-10-19T21:54:26.732Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.732Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: pipelining replication: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" testing.go:149: cluster: 2021-10-19T21:54:26.837Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.732832098 +0000 UTC m=+7.857858259 (1 nodes), 1.542µs from start of poll, 293.270355ms from cluster start. Timeout at 2021-10-19 21:54:26.837921864 +0000 UTC m=+7.962948011, 105.089752ms after stability testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.838Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: aborting pipeline replication: peer="{Voter server-267edc5a-eeec-9afa-3668-0d56dca9d72d 267edc5a-eeec-9afa-3668-0d56dca9d72d}" testing.go:149: server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: 2021-10-19T21:54:26.838Z [INFO] server-dd8ec00e-4cb0-bfa4-3d21-5fb2803f2485: aborting pipeline replication: peer="{Voter server-4c3ee407-43b8-039b-240c-1f376cf7c6d3 4c3ee407-43b8-039b-240c-1f376cf7c6d3}" === RUN TestRaft_RemoveFollower testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.841Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.842Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering follower state: follower="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Follower]" leader= testLoggerAdapter verbose: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:26.842Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testLoggerAdapter verbose: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:26.842Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: entering follower state: follower="Node at ac56bd4e-dced-78da-130c-bea36ebff0ee [Follower]" leader= testLoggerAdapter verbose: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:26.842Z [INFO] server-c46dd2a9-c44b-5505-943e-b486fe226f76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:26.843Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:26.843Z [INFO] server-c46dd2a9-c44b-5505-943e-b486fe226f76: entering follower state: follower="Node at c46dd2a9-c44b-5505-943e-b486fe226f76 [Follower]" leader= testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [WARN] server-8ca948da-ed6d-91af-d283-965d7133a144: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering candidate state: node="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Candidate]" term=2 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: election won: tally=2 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering leader state: leader="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Leader]" testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: added peer, starting replication: peer=server-ac56bd4e-dced-78da-130c-bea36ebff0ee testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: added peer, starting replication: peer=server-c46dd2a9-c44b-5505-943e-b486fe226f76 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: pipelining replication: peer="{Voter server-c46dd2a9-c44b-5505-943e-b486fe226f76 c46dd2a9-c44b-5505-943e-b486fe226f76}" testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: pipelining replication: peer="{Voter server-ac56bd4e-dced-78da-130c-bea36ebff0ee ac56bd4e-dced-78da-130c-bea36ebff0ee}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.010Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.905533984 +0000 UTC m=+8.030560112 (1 nodes), 62.218083ms from start of poll, 64.000915ms from cluster start. Timeout at 2021-10-19 21:54:27.010615685 +0000 UTC m=+8.135641817, 105.081705ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.015Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.121Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.015943701 +0000 UTC m=+8.140969825 (2 nodes), 1.624µs from start of poll, 174.410628ms from cluster start. Timeout at 2021-10-19 21:54:27.121057108 +0000 UTC m=+8.246083276, 105.113451ms after stability testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: updating configuration: command=RemoveServer server-id=server-ac56bd4e-dced-78da-130c-bea36ebff0ee server-addr= servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: removed peer, stopping replication: peer=server-ac56bd4e-dced-78da-130c-bea36ebff0ee last-index=3 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: aborting pipeline replication: peer="{Voter server-ac56bd4e-dced-78da-130c-bea36ebff0ee ac56bd4e-dced-78da-130c-bea36ebff0ee}" testLoggerAdapter verbose: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:27.180Z [WARN] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: heartbeat timeout reached, starting election: last-leader=8ca948da-ed6d-91af-d283-965d7133a144 testLoggerAdapter verbose: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:27.180Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: entering candidate state: node="Node at ac56bd4e-dced-78da-130c-bea36ebff0ee [Candidate]" term=3 testLoggerAdapter verbose: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:27.180Z [WARN] server-c46dd2a9-c44b-5505-943e-b486fe226f76: rejecting vote request since we have a leader: from=ac56bd4e-dced-78da-130c-bea36ebff0ee leader=8ca948da-ed6d-91af-d283-965d7133a144 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.180Z [WARN] server-8ca948da-ed6d-91af-d283-965d7133a144: rejecting vote request since we have a leader: from=ac56bd4e-dced-78da-130c-bea36ebff0ee leader=8ca948da-ed6d-91af-d283-965d7133a144 testLoggerAdapter verbose: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.227Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: aborting pipeline replication: peer="{Voter server-c46dd2a9-c44b-5505-943e-b486fe226f76 c46dd2a9-c44b-5505-943e-b486fe226f76}" --- PASS: TestRaft_RemoveFollower (0.40s) testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.841Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.842Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering follower state: follower="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Follower]" leader= testing.go:149: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:26.842Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testing.go:149: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:26.842Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: entering follower state: follower="Node at ac56bd4e-dced-78da-130c-bea36ebff0ee [Follower]" leader= testing.go:149: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:26.842Z [INFO] server-c46dd2a9-c44b-5505-943e-b486fe226f76: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-ac56bd4e-dced-78da-130c-bea36ebff0ee Address:ac56bd4e-dced-78da-130c-bea36ebff0ee} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testing.go:149: cluster: 2021-10-19T21:54:26.843Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:26.843Z [INFO] server-c46dd2a9-c44b-5505-943e-b486fe226f76: entering follower state: follower="Node at c46dd2a9-c44b-5505-943e-b486fe226f76 [Follower]" leader= testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [WARN] server-8ca948da-ed6d-91af-d283-965d7133a144: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering candidate state: node="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Candidate]" term=2 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: election won: tally=2 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: entering leader state: leader="Node at 8ca948da-ed6d-91af-d283-965d7133a144 [Leader]" testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: added peer, starting replication: peer=server-ac56bd4e-dced-78da-130c-bea36ebff0ee testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: added peer, starting replication: peer=server-c46dd2a9-c44b-5505-943e-b486fe226f76 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: pipelining replication: peer="{Voter server-c46dd2a9-c44b-5505-943e-b486fe226f76 c46dd2a9-c44b-5505-943e-b486fe226f76}" testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:26.905Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: pipelining replication: peer="{Voter server-ac56bd4e-dced-78da-130c-bea36ebff0ee ac56bd4e-dced-78da-130c-bea36ebff0ee}" testing.go:149: cluster: 2021-10-19T21:54:27.010Z [INFO] stable state for Leader reached at 2021-10-19 21:54:26.905533984 +0000 UTC m=+8.030560112 (1 nodes), 62.218083ms from start of poll, 64.000915ms from cluster start. Timeout at 2021-10-19 21:54:27.010615685 +0000 UTC m=+8.135641817, 105.081705ms after stability testing.go:149: cluster: 2021-10-19T21:54:27.015Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:27.121Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.015943701 +0000 UTC m=+8.140969825 (2 nodes), 1.624µs from start of poll, 174.410628ms from cluster start. Timeout at 2021-10-19 21:54:27.121057108 +0000 UTC m=+8.246083276, 105.113451ms after stability testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: updating configuration: command=RemoveServer server-id=server-ac56bd4e-dced-78da-130c-bea36ebff0ee server-addr= servers="[{Suffrage:Voter ID:server-8ca948da-ed6d-91af-d283-965d7133a144 Address:8ca948da-ed6d-91af-d283-965d7133a144} {Suffrage:Voter ID:server-c46dd2a9-c44b-5505-943e-b486fe226f76 Address:c46dd2a9-c44b-5505-943e-b486fe226f76}]" testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: removed peer, stopping replication: peer=server-ac56bd4e-dced-78da-130c-bea36ebff0ee last-index=3 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.121Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: aborting pipeline replication: peer="{Voter server-ac56bd4e-dced-78da-130c-bea36ebff0ee ac56bd4e-dced-78da-130c-bea36ebff0ee}" testing.go:149: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:27.180Z [WARN] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: heartbeat timeout reached, starting election: last-leader=8ca948da-ed6d-91af-d283-965d7133a144 testing.go:149: server-ac56bd4e-dced-78da-130c-bea36ebff0ee: 2021-10-19T21:54:27.180Z [INFO] server-ac56bd4e-dced-78da-130c-bea36ebff0ee: entering candidate state: node="Node at ac56bd4e-dced-78da-130c-bea36ebff0ee [Candidate]" term=3 testing.go:149: server-c46dd2a9-c44b-5505-943e-b486fe226f76: 2021-10-19T21:54:27.180Z [WARN] server-c46dd2a9-c44b-5505-943e-b486fe226f76: rejecting vote request since we have a leader: from=ac56bd4e-dced-78da-130c-bea36ebff0ee leader=8ca948da-ed6d-91af-d283-965d7133a144 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.180Z [WARN] server-8ca948da-ed6d-91af-d283-965d7133a144: rejecting vote request since we have a leader: from=ac56bd4e-dced-78da-130c-bea36ebff0ee leader=8ca948da-ed6d-91af-d283-965d7133a144 testing.go:149: server-8ca948da-ed6d-91af-d283-965d7133a144: 2021-10-19T21:54:27.227Z [INFO] server-8ca948da-ed6d-91af-d283-965d7133a144: aborting pipeline replication: peer="{Voter server-c46dd2a9-c44b-5505-943e-b486fe226f76 c46dd2a9-c44b-5505-943e-b486fe226f76}" === RUN TestRaft_RemoveLeader testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.236Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.236Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering follower state: follower="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Follower]" leader= testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.237Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.237Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering follower state: follower="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Follower]" leader= testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.237Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.237Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.237Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering follower state: follower="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Follower]" leader= testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [WARN] server-0f2462be-fe12-992b-ed05-3ca22752cadb: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering candidate state: node="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Candidate]" term=2 testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: election won: tally=2 testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering leader state: leader="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Leader]" testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: added peer, starting replication: peer=server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: added peer, starting replication: peer=server-986c1927-a6c8-749e-089f-eb7f3fc5823d testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: pipelining replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: pipelining replication: peer="{Voter server-986c1927-a6c8-749e-089f-eb7f3fc5823d 986c1927-a6c8-749e-089f-eb7f3fc5823d}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.407Z [INFO] stable state for Leader reached at 2021-10-19 21:54:27.302407291 +0000 UTC m=+8.427433427 (1 nodes), 65.001966ms from start of poll, 66.057035ms from cluster start. Timeout at 2021-10-19 21:54:27.407482507 +0000 UTC m=+8.532508650, 105.075223ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.412Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.518Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.412898935 +0000 UTC m=+8.537925084 (2 nodes), 1.839µs from start of poll, 176.548692ms from cluster start. Timeout at 2021-10-19 21:54:27.518015574 +0000 UTC m=+8.643041761, 105.116677ms after stability testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: updating configuration: command=RemoveServer server-id=server-0f2462be-fe12-992b-ed05-3ca22752cadb server-addr= servers="[{Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: removed ourself, shutting down testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: aborting pipeline replication: peer="{Voter server-986c1927-a6c8-749e-089f-eb7f3fc5823d 986c1927-a6c8-749e-089f-eb7f3fc5823d}" testLoggerAdapter verbose: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: aborting pipeline replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.585Z [WARN] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: heartbeat timeout reached, starting election: last-leader=0f2462be-fe12-992b-ed05-3ca22752cadb testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.585Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering candidate state: node="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Candidate]" term=3 testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.585Z [WARN] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: rejecting vote request since we have a leader: from=986c1927-a6c8-749e-089f-eb7f3fc5823d leader=0f2462be-fe12-992b-ed05-3ca22752cadb testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.603Z [WARN] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: heartbeat timeout reached, starting election: last-leader=0f2462be-fe12-992b-ed05-3ca22752cadb testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.603Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering candidate state: node="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Candidate]" term=3 testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.603Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [WARN] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: Election timeout reached, restarting election testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering candidate state: node="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Candidate]" term=4 testLoggerAdapter verbose: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.676Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering follower state: follower="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Follower]" leader= testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: election won: tally=2 testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering leader state: leader="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Leader]" testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: added peer, starting replication: peer=server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: pipelining replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.729Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.834Z [INFO] stable state for Leader reached at 2021-10-19 21:54:27.729222731 +0000 UTC m=+8.854248862 (1 nodes), 62.303µs from start of poll, 492.87247ms from cluster start. Timeout at 2021-10-19 21:54:27.834281013 +0000 UTC m=+8.959307148, 105.058286ms after stability testLoggerAdapter verbose: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.834Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: aborting pipeline replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" --- PASS: TestRaft_RemoveLeader (0.60s) testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.236Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.236Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering follower state: follower="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Follower]" leader= testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.237Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.237Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering follower state: follower="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Follower]" leader= testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.237Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f2462be-fe12-992b-ed05-3ca22752cadb Address:0f2462be-fe12-992b-ed05-3ca22752cadb} {Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testing.go:149: cluster: 2021-10-19T21:54:27.237Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.237Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering follower state: follower="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Follower]" leader= testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [WARN] server-0f2462be-fe12-992b-ed05-3ca22752cadb: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering candidate state: node="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Candidate]" term=2 testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: election won: tally=2 testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: entering leader state: leader="Node at 0f2462be-fe12-992b-ed05-3ca22752cadb [Leader]" testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: added peer, starting replication: peer=server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: added peer, starting replication: peer=server-986c1927-a6c8-749e-089f-eb7f3fc5823d testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: pipelining replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.302Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: pipelining replication: peer="{Voter server-986c1927-a6c8-749e-089f-eb7f3fc5823d 986c1927-a6c8-749e-089f-eb7f3fc5823d}" testing.go:149: cluster: 2021-10-19T21:54:27.407Z [INFO] stable state for Leader reached at 2021-10-19 21:54:27.302407291 +0000 UTC m=+8.427433427 (1 nodes), 65.001966ms from start of poll, 66.057035ms from cluster start. Timeout at 2021-10-19 21:54:27.407482507 +0000 UTC m=+8.532508650, 105.075223ms after stability testing.go:149: cluster: 2021-10-19T21:54:27.412Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:27.518Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.412898935 +0000 UTC m=+8.537925084 (2 nodes), 1.839µs from start of poll, 176.548692ms from cluster start. Timeout at 2021-10-19 21:54:27.518015574 +0000 UTC m=+8.643041761, 105.116677ms after stability testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: updating configuration: command=RemoveServer server-id=server-0f2462be-fe12-992b-ed05-3ca22752cadb server-addr= servers="[{Suffrage:Voter ID:server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 Address:e8ff1171-b867-d0af-7cae-b0a10eb9de23} {Suffrage:Voter ID:server-986c1927-a6c8-749e-089f-eb7f3fc5823d Address:986c1927-a6c8-749e-089f-eb7f3fc5823d}]" testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: removed ourself, shutting down testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: aborting pipeline replication: peer="{Voter server-986c1927-a6c8-749e-089f-eb7f3fc5823d 986c1927-a6c8-749e-089f-eb7f3fc5823d}" testing.go:149: server-0f2462be-fe12-992b-ed05-3ca22752cadb: 2021-10-19T21:54:27.518Z [INFO] server-0f2462be-fe12-992b-ed05-3ca22752cadb: aborting pipeline replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.585Z [WARN] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: heartbeat timeout reached, starting election: last-leader=0f2462be-fe12-992b-ed05-3ca22752cadb testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.585Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering candidate state: node="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Candidate]" term=3 testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.585Z [WARN] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: rejecting vote request since we have a leader: from=986c1927-a6c8-749e-089f-eb7f3fc5823d leader=0f2462be-fe12-992b-ed05-3ca22752cadb testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.603Z [WARN] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: heartbeat timeout reached, starting election: last-leader=0f2462be-fe12-992b-ed05-3ca22752cadb testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.603Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering candidate state: node="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Candidate]" term=3 testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.603Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: duplicate requestVote for same term: term=3 testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [WARN] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: Election timeout reached, restarting election testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering candidate state: node="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Candidate]" term=4 testing.go:149: server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: 2021-10-19T21:54:27.676Z [INFO] server-e8ff1171-b867-d0af-7cae-b0a10eb9de23: entering follower state: follower="Node at e8ff1171-b867-d0af-7cae-b0a10eb9de23 [Follower]" leader= testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: election won: tally=2 testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: entering leader state: leader="Node at 986c1927-a6c8-749e-089f-eb7f3fc5823d [Leader]" testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: added peer, starting replication: peer=server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.676Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: pipelining replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" testing.go:149: cluster: 2021-10-19T21:54:27.729Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:27.834Z [INFO] stable state for Leader reached at 2021-10-19 21:54:27.729222731 +0000 UTC m=+8.854248862 (1 nodes), 62.303µs from start of poll, 492.87247ms from cluster start. Timeout at 2021-10-19 21:54:27.834281013 +0000 UTC m=+8.959307148, 105.058286ms after stability testing.go:149: server-986c1927-a6c8-749e-089f-eb7f3fc5823d: 2021-10-19T21:54:27.834Z [INFO] server-986c1927-a6c8-749e-089f-eb7f3fc5823d: aborting pipeline replication: peer="{Voter server-e8ff1171-b867-d0af-7cae-b0a10eb9de23 e8ff1171-b867-d0af-7cae-b0a10eb9de23}" === RUN TestRaft_RemoveLeader_NoShutdown testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:27.838Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:27.838Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering follower state: follower="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Follower]" leader= testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.839Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.839Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering follower state: follower="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Follower]" leader= testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:27.839Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:27.839Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:27.839Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering follower state: follower="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Follower]" leader= testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [WARN] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering candidate state: node="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Candidate]" term=2 testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: election won: tally=2 testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering leader state: leader="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Leader]" testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: added peer, starting replication: peer=server-b7b2b7b7-03e6-071f-c043-6540a42576bd testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: added peer, starting replication: peer=server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: pipelining replication: peer="{Voter server-b7b2b7b7-03e6-071f-c043-6540a42576bd b7b2b7b7-03e6-071f-c043-6540a42576bd}" testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: pipelining replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.003Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.898826057 +0000 UTC m=+9.023852189 (2 nodes), 58.829833ms from start of poll, 60.863535ms from cluster start. Timeout at 2021-10-19 21:54:28.003899332 +0000 UTC m=+9.128925466, 105.073277ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.004Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.109Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.004104598 +0000 UTC m=+9.129130745 (1 nodes), 1.463µs from start of poll, 166.142091ms from cluster start. Timeout at 2021-10-19 21:54:28.109179501 +0000 UTC m=+9.234205641, 105.074896ms after stability testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: updating configuration: command=RemoveServer server-id=server-9c23a996-c7f4-e752-62ad-32a9d5260f46 server-addr= servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: removed ourself, transitioning to follower testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering follower state: follower="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Follower]" leader= testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: aborting pipeline replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: aborting pipeline replication: peer="{Voter server-b7b2b7b7-03e6-071f-c043-6540a42576bd b7b2b7b7-03e6-071f-c043-6540a42576bd}" testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.177Z [WARN] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: heartbeat timeout reached, starting election: last-leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.177Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering candidate state: node="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Candidate]" term=3 testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.177Z [WARN] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: rejecting vote request since we have a leader: from=b7b2b7b7-03e6-071f-c043-6540a42576bd leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.192Z [WARN] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: heartbeat timeout reached, starting election: last-leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.192Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering candidate state: node="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Candidate]" term=3 testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.192Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.203Z [WARN] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: not part of stable configuration, aborting election testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.217Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [WARN] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: Election timeout reached, restarting election testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering candidate state: node="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Candidate]" term=4 testLoggerAdapter verbose: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.265Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering follower state: follower="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Follower]" leader= testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: election won: tally=2 testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering leader state: leader="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Leader]" testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: added peer, starting replication: peer=server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.266Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: pipelining replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.370Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.26580115 +0000 UTC m=+9.390827304 (1 nodes), 48.656112ms from start of poll, 427.83865ms from cluster start. Timeout at 2021-10-19 21:54:28.370896992 +0000 UTC m=+9.495923143, 105.095839ms after stability testLoggerAdapter verbose: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.476Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: aborting pipeline replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" --- PASS: TestRaft_RemoveLeader_NoShutdown (0.64s) testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:27.838Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:27.838Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering follower state: follower="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Follower]" leader= testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.839Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.839Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering follower state: follower="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Follower]" leader= testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:27.839Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-9c23a996-c7f4-e752-62ad-32a9d5260f46 Address:9c23a996-c7f4-e752-62ad-32a9d5260f46} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testing.go:149: cluster: 2021-10-19T21:54:27.839Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:27.839Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering follower state: follower="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Follower]" leader= testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [WARN] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering candidate state: node="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Candidate]" term=2 testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: election won: tally=2 testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering leader state: leader="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Leader]" testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: added peer, starting replication: peer=server-b7b2b7b7-03e6-071f-c043-6540a42576bd testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: added peer, starting replication: peer=server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: pipelining replication: peer="{Voter server-b7b2b7b7-03e6-071f-c043-6540a42576bd b7b2b7b7-03e6-071f-c043-6540a42576bd}" testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:27.898Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: pipelining replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testing.go:149: cluster: 2021-10-19T21:54:28.003Z [INFO] stable state for Follower reached at 2021-10-19 21:54:27.898826057 +0000 UTC m=+9.023852189 (2 nodes), 58.829833ms from start of poll, 60.863535ms from cluster start. Timeout at 2021-10-19 21:54:28.003899332 +0000 UTC m=+9.128925466, 105.073277ms after stability testing.go:149: cluster: 2021-10-19T21:54:28.004Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:28.109Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.004104598 +0000 UTC m=+9.129130745 (1 nodes), 1.463µs from start of poll, 166.142091ms from cluster start. Timeout at 2021-10-19 21:54:28.109179501 +0000 UTC m=+9.234205641, 105.074896ms after stability testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: updating configuration: command=RemoveServer server-id=server-9c23a996-c7f4-e752-62ad-32a9d5260f46 server-addr= servers="[{Suffrage:Voter ID:server-b7b2b7b7-03e6-071f-c043-6540a42576bd Address:b7b2b7b7-03e6-071f-c043-6540a42576bd} {Suffrage:Voter ID:server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e Address:530e1268-5c1f-e15e-b537-d6abaa7c5c7e}]" testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: removed ourself, transitioning to follower testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: entering follower state: follower="Node at 9c23a996-c7f4-e752-62ad-32a9d5260f46 [Follower]" leader= testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: aborting pipeline replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.111Z [INFO] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: aborting pipeline replication: peer="{Voter server-b7b2b7b7-03e6-071f-c043-6540a42576bd b7b2b7b7-03e6-071f-c043-6540a42576bd}" testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.177Z [WARN] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: heartbeat timeout reached, starting election: last-leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.177Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering candidate state: node="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Candidate]" term=3 testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.177Z [WARN] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: rejecting vote request since we have a leader: from=b7b2b7b7-03e6-071f-c043-6540a42576bd leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.192Z [WARN] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: heartbeat timeout reached, starting election: last-leader=9c23a996-c7f4-e752-62ad-32a9d5260f46 testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.192Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering candidate state: node="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Candidate]" term=3 testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.192Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: duplicate requestVote for same term: term=3 testing.go:149: server-9c23a996-c7f4-e752-62ad-32a9d5260f46: 2021-10-19T21:54:28.203Z [WARN] server-9c23a996-c7f4-e752-62ad-32a9d5260f46: not part of stable configuration, aborting election testing.go:149: cluster: 2021-10-19T21:54:28.217Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [WARN] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: Election timeout reached, restarting election testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering candidate state: node="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Candidate]" term=4 testing.go:149: server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: 2021-10-19T21:54:28.265Z [INFO] server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e: entering follower state: follower="Node at 530e1268-5c1f-e15e-b537-d6abaa7c5c7e [Follower]" leader= testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: election won: tally=2 testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: entering leader state: leader="Node at b7b2b7b7-03e6-071f-c043-6540a42576bd [Leader]" testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.265Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: added peer, starting replication: peer=server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.266Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: pipelining replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" testing.go:149: cluster: 2021-10-19T21:54:28.370Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.26580115 +0000 UTC m=+9.390827304 (1 nodes), 48.656112ms from start of poll, 427.83865ms from cluster start. Timeout at 2021-10-19 21:54:28.370896992 +0000 UTC m=+9.495923143, 105.095839ms after stability testing.go:149: server-b7b2b7b7-03e6-071f-c043-6540a42576bd: 2021-10-19T21:54:28.476Z [INFO] server-b7b2b7b7-03e6-071f-c043-6540a42576bd: aborting pipeline replication: peer="{Voter server-530e1268-5c1f-e15e-b537-d6abaa7c5c7e 530e1268-5c1f-e15e-b537-d6abaa7c5c7e}" === RUN TestRaft_RemoveFollower_SplitCluster testLoggerAdapter verbose: server-8944fd8f-1146-0c84-c88f-58b3865ec38a: 2021-10-19T21:54:28.478Z [INFO] server-8944fd8f-1146-0c84-c88f-58b3865ec38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testLoggerAdapter verbose: server-8944fd8f-1146-0c84-c88f-58b3865ec38a: 2021-10-19T21:54:28.478Z [INFO] server-8944fd8f-1146-0c84-c88f-58b3865ec38a: entering follower state: follower="Node at 8944fd8f-1146-0c84-c88f-58b3865ec38a [Follower]" leader= testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.478Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.478Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering follower state: follower="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Follower]" leader= testLoggerAdapter verbose: server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: 2021-10-19T21:54:28.478Z [INFO] server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testLoggerAdapter verbose: server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: 2021-10-19T21:54:28.478Z [INFO] server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: entering follower state: follower="Node at c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 [Follower]" leader= testLoggerAdapter verbose: server-636059e6-8d9d-82d3-f9aa-7a1005e60737: 2021-10-19T21:54:28.479Z [INFO] server-636059e6-8d9d-82d3-f9aa-7a1005e60737: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.479Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-636059e6-8d9d-82d3-f9aa-7a1005e60737: 2021-10-19T21:54:28.479Z [INFO] server-636059e6-8d9d-82d3-f9aa-7a1005e60737: entering follower state: follower="Node at 636059e6-8d9d-82d3-f9aa-7a1005e60737 [Follower]" leader= testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering candidate state: node="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Candidate]" term=2 testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: election won: tally=3 testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering leader state: leader="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Leader]" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-8944fd8f-1146-0c84-c88f-58b3865ec38a testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-636059e6-8d9d-82d3-f9aa-7a1005e60737 636059e6-8d9d-82d3-f9aa-7a1005e60737}" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.653Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.548161819 +0000 UTC m=+9.673187958 (1 nodes), 68.962216ms from start of poll, 69.731601ms from cluster start. Timeout at 2021-10-19 21:54:28.653245098 +0000 UTC m=+9.778271255, 105.083297ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.758Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.863Z [INFO] stable state for Follower reached at 2021-10-19 21:54:28.758803428 +0000 UTC m=+9.883829574 (3 nodes), 2.449µs from start of poll, 280.373217ms from cluster start. Timeout at 2021-10-19 21:54:28.86386643 +0000 UTC m=+9.988892575, 105.063001ms after stability testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: updating configuration: command=RemoveServer server-id=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 server-addr= servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}]" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: removed peer, stopping replication: peer=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 last-index=3 testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to pipeline appendEntries: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="append pipeline closed" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to pipeline appendEntries: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="append pipeline closed" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-636059e6-8d9d-82d3-f9aa-7a1005e60737 636059e6-8d9d-82d3-f9aa-7a1005e60737}" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.868Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.870Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.872Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.880Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.883Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.889Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.890Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.895Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.899Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-8944fd8f-1146-0c84-c88f-58b3865ec38a time=50.105415ms testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-8944fd8f-1146-0c84-c88f-58b3865ec38a time=60.348143ms testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 time=58.710784ms testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering follower state: follower="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Follower]" leader= testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.919Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testLoggerAdapter verbose: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.919Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" --- PASS: TestRaft_RemoveFollower_SplitCluster (0.44s) testing.go:149: server-8944fd8f-1146-0c84-c88f-58b3865ec38a: 2021-10-19T21:54:28.478Z [INFO] server-8944fd8f-1146-0c84-c88f-58b3865ec38a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testing.go:149: server-8944fd8f-1146-0c84-c88f-58b3865ec38a: 2021-10-19T21:54:28.478Z [INFO] server-8944fd8f-1146-0c84-c88f-58b3865ec38a: entering follower state: follower="Node at 8944fd8f-1146-0c84-c88f-58b3865ec38a [Follower]" leader= testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.478Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.478Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering follower state: follower="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Follower]" leader= testing.go:149: server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: 2021-10-19T21:54:28.478Z [INFO] server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testing.go:149: server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: 2021-10-19T21:54:28.478Z [INFO] server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0: entering follower state: follower="Node at c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 [Follower]" leader= testing.go:149: server-636059e6-8d9d-82d3-f9aa-7a1005e60737: 2021-10-19T21:54:28.479Z [INFO] server-636059e6-8d9d-82d3-f9aa-7a1005e60737: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0} {Suffrage:Voter ID:server-636059e6-8d9d-82d3-f9aa-7a1005e60737 Address:636059e6-8d9d-82d3-f9aa-7a1005e60737}]" testing.go:149: cluster: 2021-10-19T21:54:28.479Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-636059e6-8d9d-82d3-f9aa-7a1005e60737: 2021-10-19T21:54:28.479Z [INFO] server-636059e6-8d9d-82d3-f9aa-7a1005e60737: entering follower state: follower="Node at 636059e6-8d9d-82d3-f9aa-7a1005e60737 [Follower]" leader= testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering candidate state: node="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Candidate]" term=2 testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: election won: tally=3 testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering leader state: leader="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Leader]" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-8944fd8f-1146-0c84-c88f-58b3865ec38a testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.547Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: added peer, starting replication: peer=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-636059e6-8d9d-82d3-f9aa-7a1005e60737 636059e6-8d9d-82d3-f9aa-7a1005e60737}" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.548Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: pipelining replication: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" testing.go:149: cluster: 2021-10-19T21:54:28.653Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.548161819 +0000 UTC m=+9.673187958 (1 nodes), 68.962216ms from start of poll, 69.731601ms from cluster start. Timeout at 2021-10-19 21:54:28.653245098 +0000 UTC m=+9.778271255, 105.083297ms after stability testing.go:149: cluster: 2021-10-19T21:54:28.758Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:28.863Z [INFO] stable state for Follower reached at 2021-10-19 21:54:28.758803428 +0000 UTC m=+9.883829574 (3 nodes), 2.449µs from start of poll, 280.373217ms from cluster start. Timeout at 2021-10-19 21:54:28.86386643 +0000 UTC m=+9.988892575, 105.063001ms after stability testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: updating configuration: command=RemoveServer server-id=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 server-addr= servers="[{Suffrage:Voter ID:server-8944fd8f-1146-0c84-c88f-58b3865ec38a Address:8944fd8f-1146-0c84-c88f-58b3865ec38a} {Suffrage:Voter ID:server-c3499696-41d0-3532-2727-3bd260b3c84a Address:c3499696-41d0-3532-2727-3bd260b3c84a} {Suffrage:Voter ID:server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 Address:c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}]" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: removed peer, stopping replication: peer=server-636059e6-8d9d-82d3-f9aa-7a1005e60737 last-index=3 testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to pipeline appendEntries: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="append pipeline closed" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to pipeline appendEntries: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="append pipeline closed" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.864Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: aborting pipeline replication: peer="{Voter server-636059e6-8d9d-82d3-f9aa-7a1005e60737 636059e6-8d9d-82d3-f9aa-7a1005e60737}" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.868Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.870Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.872Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.880Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.883Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.889Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.890Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.895Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=8944fd8f-1146-0c84-c88f-58b3865ec38a error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.899Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to heartbeat to: peer=c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.908Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-8944fd8f-1146-0c84-c88f-58b3865ec38a time=50.105415ms testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-8944fd8f-1146-0c84-c88f-58b3865ec38a time=60.348143ms testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact: server-id=server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 time=58.710784ms testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [WARN] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to contact quorum of nodes, stepping down testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.918Z [INFO] server-c3499696-41d0-3532-2727-3bd260b3c84a: entering follower state: follower="Node at c3499696-41d0-3532-2727-3bd260b3c84a [Follower]" leader= testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.919Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-c464a0b7-187e-8d6b-7b57-0d8de7ad46b0 c464a0b7-187e-8d6b-7b57-0d8de7ad46b0}" error="failed to connect to peer: c464a0b7-187e-8d6b-7b57-0d8de7ad46b0" testing.go:149: server-c3499696-41d0-3532-2727-3bd260b3c84a: 2021-10-19T21:54:28.919Z [ERROR] server-c3499696-41d0-3532-2727-3bd260b3c84a: failed to appendEntries to: peer="{Voter server-8944fd8f-1146-0c84-c88f-58b3865ec38a 8944fd8f-1146-0c84-c88f-58b3865ec38a}" error="failed to connect to peer: 8944fd8f-1146-0c84-c88f-58b3865ec38a" === RUN TestRaft_AddKnownPeer testLoggerAdapter verbose: server-3fdcb9e3-091c-4ff2-1871-363a3af84176: 2021-10-19T21:54:28.924Z [INFO] server-3fdcb9e3-091c-4ff2-1871-363a3af84176: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testLoggerAdapter verbose: server-3fdcb9e3-091c-4ff2-1871-363a3af84176: 2021-10-19T21:54:28.925Z [INFO] server-3fdcb9e3-091c-4ff2-1871-363a3af84176: entering follower state: follower="Node at 3fdcb9e3-091c-4ff2-1871-363a3af84176 [Follower]" leader= testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.926Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.926Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering follower state: follower="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Follower]" leader= testLoggerAdapter verbose: server-86315cec-57cd-e13a-42b5-a410b8bbbd60: 2021-10-19T21:54:28.926Z [INFO] server-86315cec-57cd-e13a-42b5-a410b8bbbd60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:28.926Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-86315cec-57cd-e13a-42b5-a410b8bbbd60: 2021-10-19T21:54:28.926Z [INFO] server-86315cec-57cd-e13a-42b5-a410b8bbbd60: entering follower state: follower="Node at 86315cec-57cd-e13a-42b5-a410b8bbbd60 [Follower]" leader= testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [WARN] server-13a54269-1c33-4f68-6842-f029e9574244: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering candidate state: node="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Candidate]" term=2 testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: election won: tally=2 testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering leader state: leader="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Leader]" testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: added peer, starting replication: peer=server-3fdcb9e3-091c-4ff2-1871-363a3af84176 testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: added peer, starting replication: peer=server-86315cec-57cd-e13a-42b5-a410b8bbbd60 testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: pipelining replication: peer="{Voter server-86315cec-57cd-e13a-42b5-a410b8bbbd60 86315cec-57cd-e13a-42b5-a410b8bbbd60}" testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: pipelining replication: peer="{Voter server-3fdcb9e3-091c-4ff2-1871-363a3af84176 3fdcb9e3-091c-4ff2-1871-363a3af84176}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.096Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.990963263 +0000 UTC m=+10.115989411 (1 nodes), 64.384128ms from start of poll, 66.594429ms from cluster start. Timeout at 2021-10-19 21:54:29.096056383 +0000 UTC m=+10.221082531, 105.09312ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.096Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.201Z [INFO] stable state for Follower reached at 2021-10-19 21:54:29.096239317 +0000 UTC m=+10.221265456 (2 nodes), 1.266µs from start of poll, 171.870474ms from cluster start. Timeout at 2021-10-19 21:54:29.201308763 +0000 UTC m=+10.326334911, 105.069455ms after stability testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: updating configuration: command=AddStaging server-id=server-3fdcb9e3-091c-4ff2-1871-363a3af84176 server-addr=3fdcb9e3-091c-4ff2-1871-363a3af84176 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: aborting pipeline replication: peer="{Voter server-86315cec-57cd-e13a-42b5-a410b8bbbd60 86315cec-57cd-e13a-42b5-a410b8bbbd60}" testLoggerAdapter verbose: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: aborting pipeline replication: peer="{Voter server-3fdcb9e3-091c-4ff2-1871-363a3af84176 3fdcb9e3-091c-4ff2-1871-363a3af84176}" --- PASS: TestRaft_AddKnownPeer (0.28s) testing.go:149: server-3fdcb9e3-091c-4ff2-1871-363a3af84176: 2021-10-19T21:54:28.924Z [INFO] server-3fdcb9e3-091c-4ff2-1871-363a3af84176: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testing.go:149: server-3fdcb9e3-091c-4ff2-1871-363a3af84176: 2021-10-19T21:54:28.925Z [INFO] server-3fdcb9e3-091c-4ff2-1871-363a3af84176: entering follower state: follower="Node at 3fdcb9e3-091c-4ff2-1871-363a3af84176 [Follower]" leader= testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.926Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.926Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering follower state: follower="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Follower]" leader= testing.go:149: server-86315cec-57cd-e13a-42b5-a410b8bbbd60: 2021-10-19T21:54:28.926Z [INFO] server-86315cec-57cd-e13a-42b5-a410b8bbbd60: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testing.go:149: cluster: 2021-10-19T21:54:28.926Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-86315cec-57cd-e13a-42b5-a410b8bbbd60: 2021-10-19T21:54:28.926Z [INFO] server-86315cec-57cd-e13a-42b5-a410b8bbbd60: entering follower state: follower="Node at 86315cec-57cd-e13a-42b5-a410b8bbbd60 [Follower]" leader= testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [WARN] server-13a54269-1c33-4f68-6842-f029e9574244: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering candidate state: node="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Candidate]" term=2 testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: election won: tally=2 testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: entering leader state: leader="Node at 13a54269-1c33-4f68-6842-f029e9574244 [Leader]" testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.990Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: added peer, starting replication: peer=server-3fdcb9e3-091c-4ff2-1871-363a3af84176 testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: added peer, starting replication: peer=server-86315cec-57cd-e13a-42b5-a410b8bbbd60 testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: pipelining replication: peer="{Voter server-86315cec-57cd-e13a-42b5-a410b8bbbd60 86315cec-57cd-e13a-42b5-a410b8bbbd60}" testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:28.991Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: pipelining replication: peer="{Voter server-3fdcb9e3-091c-4ff2-1871-363a3af84176 3fdcb9e3-091c-4ff2-1871-363a3af84176}" testing.go:149: cluster: 2021-10-19T21:54:29.096Z [INFO] stable state for Leader reached at 2021-10-19 21:54:28.990963263 +0000 UTC m=+10.115989411 (1 nodes), 64.384128ms from start of poll, 66.594429ms from cluster start. Timeout at 2021-10-19 21:54:29.096056383 +0000 UTC m=+10.221082531, 105.09312ms after stability testing.go:149: cluster: 2021-10-19T21:54:29.096Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:29.201Z [INFO] stable state for Follower reached at 2021-10-19 21:54:29.096239317 +0000 UTC m=+10.221265456 (2 nodes), 1.266µs from start of poll, 171.870474ms from cluster start. Timeout at 2021-10-19 21:54:29.201308763 +0000 UTC m=+10.326334911, 105.069455ms after stability testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: updating configuration: command=AddStaging server-id=server-3fdcb9e3-091c-4ff2-1871-363a3af84176 server-addr=3fdcb9e3-091c-4ff2-1871-363a3af84176 servers="[{Suffrage:Voter ID:server-3fdcb9e3-091c-4ff2-1871-363a3af84176 Address:3fdcb9e3-091c-4ff2-1871-363a3af84176} {Suffrage:Voter ID:server-13a54269-1c33-4f68-6842-f029e9574244 Address:13a54269-1c33-4f68-6842-f029e9574244} {Suffrage:Voter ID:server-86315cec-57cd-e13a-42b5-a410b8bbbd60 Address:86315cec-57cd-e13a-42b5-a410b8bbbd60}]" testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: aborting pipeline replication: peer="{Voter server-86315cec-57cd-e13a-42b5-a410b8bbbd60 86315cec-57cd-e13a-42b5-a410b8bbbd60}" testing.go:149: server-13a54269-1c33-4f68-6842-f029e9574244: 2021-10-19T21:54:29.201Z [INFO] server-13a54269-1c33-4f68-6842-f029e9574244: aborting pipeline replication: peer="{Voter server-3fdcb9e3-091c-4ff2-1871-363a3af84176 3fdcb9e3-091c-4ff2-1871-363a3af84176}" === RUN TestRaft_RemoveUnknownPeer testLoggerAdapter verbose: server-221dd8cb-b0a8-d378-7388-65e0531059ed: 2021-10-19T21:54:29.203Z [INFO] server-221dd8cb-b0a8-d378-7388-65e0531059ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testLoggerAdapter verbose: server-221dd8cb-b0a8-d378-7388-65e0531059ed: 2021-10-19T21:54:29.203Z [INFO] server-221dd8cb-b0a8-d378-7388-65e0531059ed: entering follower state: follower="Node at 221dd8cb-b0a8-d378-7388-65e0531059ed [Follower]" leader= testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.203Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.204Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering follower state: follower="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Follower]" leader= testLoggerAdapter verbose: server-265852e3-40d7-eb4d-bff2-445becf935bb: 2021-10-19T21:54:29.204Z [INFO] server-265852e3-40d7-eb4d-bff2-445becf935bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.204Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-265852e3-40d7-eb4d-bff2-445becf935bb: 2021-10-19T21:54:29.204Z [INFO] server-265852e3-40d7-eb4d-bff2-445becf935bb: entering follower state: follower="Node at 265852e3-40d7-eb4d-bff2-445becf935bb [Follower]" leader= testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.275Z [WARN] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.275Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering candidate state: node="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Candidate]" term=2 testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: election won: tally=2 testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering leader state: leader="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Leader]" testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: added peer, starting replication: peer=server-221dd8cb-b0a8-d378-7388-65e0531059ed testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: added peer, starting replication: peer=server-265852e3-40d7-eb4d-bff2-445becf935bb testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: pipelining replication: peer="{Voter server-221dd8cb-b0a8-d378-7388-65e0531059ed 221dd8cb-b0a8-d378-7388-65e0531059ed}" testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: pipelining replication: peer="{Voter server-265852e3-40d7-eb4d-bff2-445becf935bb 265852e3-40d7-eb4d-bff2-445becf935bb}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.381Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.276065871 +0000 UTC m=+10.401091999 (1 nodes), 71.735443ms from start of poll, 72.38138ms from cluster start. Timeout at 2021-10-19 21:54:29.38114589 +0000 UTC m=+10.506172070, 105.080071ms after stability testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: updating configuration: command=RemoveServer server-id=8c2fb92b-8b71-be5a-b576-8133f223d6ab server-addr= servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: aborting pipeline replication: peer="{Voter server-221dd8cb-b0a8-d378-7388-65e0531059ed 221dd8cb-b0a8-d378-7388-65e0531059ed}" testLoggerAdapter verbose: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: aborting pipeline replication: peer="{Voter server-265852e3-40d7-eb4d-bff2-445becf935bb 265852e3-40d7-eb4d-bff2-445becf935bb}" --- PASS: TestRaft_RemoveUnknownPeer (0.18s) testing.go:149: server-221dd8cb-b0a8-d378-7388-65e0531059ed: 2021-10-19T21:54:29.203Z [INFO] server-221dd8cb-b0a8-d378-7388-65e0531059ed: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testing.go:149: server-221dd8cb-b0a8-d378-7388-65e0531059ed: 2021-10-19T21:54:29.203Z [INFO] server-221dd8cb-b0a8-d378-7388-65e0531059ed: entering follower state: follower="Node at 221dd8cb-b0a8-d378-7388-65e0531059ed [Follower]" leader= testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.203Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.204Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering follower state: follower="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Follower]" leader= testing.go:149: server-265852e3-40d7-eb4d-bff2-445becf935bb: 2021-10-19T21:54:29.204Z [INFO] server-265852e3-40d7-eb4d-bff2-445becf935bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testing.go:149: cluster: 2021-10-19T21:54:29.204Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-265852e3-40d7-eb4d-bff2-445becf935bb: 2021-10-19T21:54:29.204Z [INFO] server-265852e3-40d7-eb4d-bff2-445becf935bb: entering follower state: follower="Node at 265852e3-40d7-eb4d-bff2-445becf935bb [Follower]" leader= testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.275Z [WARN] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.275Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering candidate state: node="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Candidate]" term=2 testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: election won: tally=2 testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: entering leader state: leader="Node at cdbbd502-4ea7-a032-f04b-11e80f4e7d05 [Leader]" testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: added peer, starting replication: peer=server-221dd8cb-b0a8-d378-7388-65e0531059ed testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: added peer, starting replication: peer=server-265852e3-40d7-eb4d-bff2-445becf935bb testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: pipelining replication: peer="{Voter server-221dd8cb-b0a8-d378-7388-65e0531059ed 221dd8cb-b0a8-d378-7388-65e0531059ed}" testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.276Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: pipelining replication: peer="{Voter server-265852e3-40d7-eb4d-bff2-445becf935bb 265852e3-40d7-eb4d-bff2-445becf935bb}" testing.go:149: cluster: 2021-10-19T21:54:29.381Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.276065871 +0000 UTC m=+10.401091999 (1 nodes), 71.735443ms from start of poll, 72.38138ms from cluster start. Timeout at 2021-10-19 21:54:29.38114589 +0000 UTC m=+10.506172070, 105.080071ms after stability testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: updating configuration: command=RemoveServer server-id=8c2fb92b-8b71-be5a-b576-8133f223d6ab server-addr= servers="[{Suffrage:Voter ID:server-221dd8cb-b0a8-d378-7388-65e0531059ed Address:221dd8cb-b0a8-d378-7388-65e0531059ed} {Suffrage:Voter ID:server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05 Address:cdbbd502-4ea7-a032-f04b-11e80f4e7d05} {Suffrage:Voter ID:server-265852e3-40d7-eb4d-bff2-445becf935bb Address:265852e3-40d7-eb4d-bff2-445becf935bb}]" testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: aborting pipeline replication: peer="{Voter server-221dd8cb-b0a8-d378-7388-65e0531059ed 221dd8cb-b0a8-d378-7388-65e0531059ed}" testing.go:149: server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: 2021-10-19T21:54:29.381Z [INFO] server-cdbbd502-4ea7-a032-f04b-11e80f4e7d05: aborting pipeline replication: peer="{Voter server-265852e3-40d7-eb4d-bff2-445becf935bb 265852e3-40d7-eb4d-bff2-445becf935bb}" === RUN TestRaft_SnapshotRestore testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.383Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-828bea3a-1739-af97-4256-de95792b1036 Address:828bea3a-1739-af97-4256-de95792b1036}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.383Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.383Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering follower state: follower="Node at 828bea3a-1739-af97-4256-de95792b1036 [Follower]" leader= testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.460Z [WARN] server-828bea3a-1739-af97-4256-de95792b1036: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering candidate state: node="Node at 828bea3a-1739-af97-4256-de95792b1036 [Candidate]" term=2 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: election won: tally=1 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering leader state: leader="Node at 828bea3a-1739-af97-4256-de95792b1036 [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.566Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.461207454 +0000 UTC m=+10.586233588 (1 nodes), 77.979348ms from start of poll, 78.187014ms from cluster start. Timeout at 2021-10-19 21:54:29.566260246 +0000 UTC m=+10.691286388, 105.0528ms after stability testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.567Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: starting snapshot up to: index=102 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: snapshot complete up to: index=102 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: restored from snapshot: id=2-102-1634680469567 testLoggerAdapter verbose: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-828bea3a-1739-af97-4256-de95792b1036 Address:828bea3a-1739-af97-4256-de95792b1036}]" --- PASS: TestRaft_SnapshotRestore (0.19s) testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.383Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-828bea3a-1739-af97-4256-de95792b1036 Address:828bea3a-1739-af97-4256-de95792b1036}]" testing.go:149: cluster: 2021-10-19T21:54:29.383Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.383Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering follower state: follower="Node at 828bea3a-1739-af97-4256-de95792b1036 [Follower]" leader= testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.460Z [WARN] server-828bea3a-1739-af97-4256-de95792b1036: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering candidate state: node="Node at 828bea3a-1739-af97-4256-de95792b1036 [Candidate]" term=2 testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: election won: tally=1 testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.461Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: entering leader state: leader="Node at 828bea3a-1739-af97-4256-de95792b1036 [Leader]" testing.go:149: cluster: 2021-10-19T21:54:29.566Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.461207454 +0000 UTC m=+10.586233588 (1 nodes), 77.979348ms from start of poll, 78.187014ms from cluster start. Timeout at 2021-10-19 21:54:29.566260246 +0000 UTC m=+10.691286388, 105.0528ms after stability testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.567Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: starting snapshot up to: index=102 testing.go:153: 2021-10-19T21:54:29.567Z [INFO] creating new snapshot: path=/tmp/raft176685085/snapshots/2-102-1634680469567.tmp testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: compacting logs: from=1 to=92 testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: snapshot complete up to: index=102 testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: restored from snapshot: id=2-102-1634680469567 testing.go:149: server-828bea3a-1739-af97-4256-de95792b1036: 2021-10-19T21:54:29.570Z [INFO] server-828bea3a-1739-af97-4256-de95792b1036: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-828bea3a-1739-af97-4256-de95792b1036 Address:828bea3a-1739-af97-4256-de95792b1036}]" === RUN TestRaft_NoRestoreOnStart testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.572Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79a90f33-6cbb-cb30-3999-6932acdb25be Address:79a90f33-6cbb-cb30-3999-6932acdb25be}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.572Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.572Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering follower state: follower="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Follower]" leader= testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [WARN] server-79a90f33-6cbb-cb30-3999-6932acdb25be: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering candidate state: node="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Candidate]" term=2 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: election won: tally=1 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering leader state: leader="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.731Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.6264562 +0000 UTC m=+10.751482325 (1 nodes), 53.893269ms from start of poll, 54.135385ms from cluster start. Timeout at 2021-10-19 21:54:29.731542501 +0000 UTC m=+10.856568656, 105.086331ms after stability testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.733Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: starting snapshot up to: index=102 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.735Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.735Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: snapshot complete up to: index=102 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.737Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79a90f33-6cbb-cb30-3999-6932acdb25be Address:79a90f33-6cbb-cb30-3999-6932acdb25be}]" --- PASS: TestRaft_NoRestoreOnStart (0.17s) testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.572Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79a90f33-6cbb-cb30-3999-6932acdb25be Address:79a90f33-6cbb-cb30-3999-6932acdb25be}]" testing.go:149: cluster: 2021-10-19T21:54:29.572Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.572Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering follower state: follower="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Follower]" leader= testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [WARN] server-79a90f33-6cbb-cb30-3999-6932acdb25be: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering candidate state: node="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Candidate]" term=2 testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: election won: tally=1 testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.626Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering leader state: leader="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Leader]" testing.go:149: cluster: 2021-10-19T21:54:29.731Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.6264562 +0000 UTC m=+10.751482325 (1 nodes), 53.893269ms from start of poll, 54.135385ms from cluster start. Timeout at 2021-10-19 21:54:29.731542501 +0000 UTC m=+10.856568656, 105.086331ms after stability testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.733Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: starting snapshot up to: index=102 testing.go:153: 2021-10-19T21:54:29.733Z [INFO] creating new snapshot: path=/tmp/raft670910039/snapshots/2-102-1634680469733.tmp testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.735Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: compacting logs: from=1 to=92 testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.735Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: snapshot complete up to: index=102 testing.go:149: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.737Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-79a90f33-6cbb-cb30-3999-6932acdb25be Address:79a90f33-6cbb-cb30-3999-6932acdb25be}]" === RUN TestRaft_SnapshotRestore_PeerChange --- SKIP: TestRaft_SnapshotRestore_PeerChange (0.00s) raft_test.go:1026: Disabled unreliable test === RUN TestRaft_AutoSnapshot --- SKIP: TestRaft_AutoSnapshot (0.00s) raft_test.go:1132: Disabled unreliable test === RUN TestRaft_UserSnapshot testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.737Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering follower state: follower="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Follower]" leader= testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.738Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e51cb729-71d9-c40c-abcd-5f416cb299bc Address:e51cb729-71d9-c40c-abcd-5f416cb299bc}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.739Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.739Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering follower state: follower="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Follower]" leader= testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.788Z [WARN] server-79a90f33-6cbb-cb30-3999-6932acdb25be: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.788Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering candidate state: node="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Candidate]" term=3 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.788Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: election won: tally=1 testLoggerAdapter verbose: server-79a90f33-6cbb-cb30-3999-6932acdb25be: 2021-10-19T21:54:29.788Z [INFO] server-79a90f33-6cbb-cb30-3999-6932acdb25be: entering leader state: leader="Node at 79a90f33-6cbb-cb30-3999-6932acdb25be [Leader]" testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [WARN] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering candidate state: node="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Candidate]" term=2 testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: election won: tally=1 testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering leader state: leader="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.911Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.806522584 +0000 UTC m=+10.931548718 (1 nodes), 67.43512ms from start of poll, 67.910966ms from cluster start. Timeout at 2021-10-19 21:54:29.911659979 +0000 UTC m=+11.036686173, 105.137455ms after stability testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.911Z [ERROR] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.912Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: starting snapshot up to: index=12 testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.917Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: compacting logs: from=1 to=2 testLoggerAdapter verbose: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.917Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: snapshot complete up to: index=12 --- PASS: TestRaft_UserSnapshot (0.18s) testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.738Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e51cb729-71d9-c40c-abcd-5f416cb299bc Address:e51cb729-71d9-c40c-abcd-5f416cb299bc}]" testing.go:149: cluster: 2021-10-19T21:54:29.739Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.739Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering follower state: follower="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Follower]" leader= testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [WARN] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering candidate state: node="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Candidate]" term=2 testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: election won: tally=1 testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.806Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: entering leader state: leader="Node at e51cb729-71d9-c40c-abcd-5f416cb299bc [Leader]" testing.go:149: cluster: 2021-10-19T21:54:29.911Z [INFO] stable state for Leader reached at 2021-10-19 21:54:29.806522584 +0000 UTC m=+10.931548718 (1 nodes), 67.43512ms from start of poll, 67.910966ms from cluster start. Timeout at 2021-10-19 21:54:29.911659979 +0000 UTC m=+11.036686173, 105.137455ms after stability testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.911Z [ERROR] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.912Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:29.912Z [INFO] creating new snapshot: path=/tmp/raft278541985/snapshots/2-12-1634680469912.tmp testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.917Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: compacting logs: from=1 to=2 testing.go:149: server-e51cb729-71d9-c40c-abcd-5f416cb299bc: 2021-10-19T21:54:29.917Z [INFO] server-e51cb729-71d9-c40c-abcd-5f416cb299bc: snapshot complete up to: index=12 === RUN TestRaft_UserRestore testLoggerAdapter verbose: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:29.921Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testLoggerAdapter verbose: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:29.921Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: entering follower state: follower="Node at 469a5b6d-8759-46c6-040e-c8b25b7c2908 [Follower]" leader= testLoggerAdapter verbose: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:29.921Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testLoggerAdapter verbose: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:29.921Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: entering follower state: follower="Node at d92b10e5-d2fd-9860-e66a-1e9725eb1579 [Follower]" leader= testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:29.922Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:29.922Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:29.922Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering follower state: follower="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Follower]" leader= testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [WARN] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering candidate state: node="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Candidate]" term=2 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: election won: tally=2 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering leader state: leader="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Leader]" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: added peer, starting replication: peer=server-469a5b6d-8759-46c6-040e-c8b25b7c2908 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: added peer, starting replication: peer=server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.105Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.000650293 +0000 UTC m=+11.125676416 (1 nodes), 78.371947ms from start of poll, 79.702081ms from cluster start. Timeout at 2021-10-19 21:54:30.105659082 +0000 UTC m=+11.230685224, 105.008808ms after stability testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.106Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: starting snapshot up to: index=12 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.109Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: snapshot complete up to: index=12 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: restored user snapshot: index=1 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.114Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.120Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.120Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.124Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:30.125Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.125Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: Installed remote snapshot testLoggerAdapter verbose: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.125Z [WARN] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: skipping application of old log: index=22 testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.125Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testLoggerAdapter verbose: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:30.125Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: Installed remote snapshot testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.125Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.136Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testLoggerAdapter verbose: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.136Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testLoggerAdapter verbose: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.138Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testLoggerAdapter verbose: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.139Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: entering follower state: follower="Node at 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 [Follower]" leader= testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.139Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testLoggerAdapter verbose: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.139Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.140Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.140Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering follower state: follower="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Follower]" leader= testLoggerAdapter verbose: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.140Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: entering follower state: follower="Node at 81228510-69dc-83c7-166d-0d5e5826bac1 [Follower]" leader= testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [WARN] server-9008c103-20bb-b91b-ec57-0716efd507f6: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering candidate state: node="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Candidate]" term=2 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: election won: tally=2 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering leader state: leader="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Leader]" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: added peer, starting replication: peer=server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: added peer, starting replication: peer=server-81228510-69dc-83c7-166d-0d5e5826bac1 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.308Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.203325515 +0000 UTC m=+11.328351656 (1 nodes), 63.269967ms from start of poll, 64.606969ms from cluster start. Timeout at 2021-10-19 21:54:30.30841024 +0000 UTC m=+11.433436400, 105.084744ms after stability testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.308Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: starting snapshot up to: index=12 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.312Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: snapshot complete up to: index=12 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.316Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: restored user snapshot: index=1 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.323Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.326Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.328Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.329Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: Installed remote snapshot testLoggerAdapter verbose: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.329Z [WARN] server-81228510-69dc-83c7-166d-0d5e5826bac1: skipping application of old log: index=22 testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.329Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testLoggerAdapter verbose: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.330Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.330Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: Installed remote snapshot testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.330Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.345Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testLoggerAdapter verbose: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.345Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testLoggerAdapter verbose: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.348Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testLoggerAdapter verbose: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.348Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: entering follower state: follower="Node at 7477a398-86af-21be-b681-8103cbb497f1 [Follower]" leader= testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.348Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.348Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering follower state: follower="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Follower]" leader= testLoggerAdapter verbose: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.348Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.348Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.348Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: entering follower state: follower="Node at c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 [Follower]" leader= testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.428Z [WARN] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering candidate state: node="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Candidate]" term=2 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: election won: tally=2 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering leader state: leader="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Leader]" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: added peer, starting replication: peer=server-7477a398-86af-21be-b681-8103cbb497f1 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: added peer, starting replication: peer=server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.534Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.429420484 +0000 UTC m=+11.554446672 (1 nodes), 80.492678ms from start of poll, 81.176712ms from cluster start. Timeout at 2021-10-19 21:54:30.534519846 +0000 UTC m=+11.659545996, 105.099324ms after stability testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.535Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: starting snapshot up to: index=12 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.539Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: snapshot complete up to: index=12 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: restored user snapshot: index=1 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.549Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.551Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: Installed remote snapshot testLoggerAdapter verbose: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [WARN] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: skipping application of old log: index=22 testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.552Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testLoggerAdapter verbose: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.554Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.554Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: Installed remote snapshot testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.554Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.573Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testLoggerAdapter verbose: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.574Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testLoggerAdapter verbose: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.576Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testLoggerAdapter verbose: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.576Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: entering follower state: follower="Node at 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 [Follower]" leader= testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.576Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.576Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering follower state: follower="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Follower]" leader= testLoggerAdapter verbose: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.576Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.577Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.577Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: entering follower state: follower="Node at c01ec779-f38f-c9ba-1968-37440a72b261 [Follower]" leader= testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.627Z [WARN] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.627Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering candidate state: node="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Candidate]" term=2 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: election won: tally=2 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering leader state: leader="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Leader]" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: added peer, starting replication: peer=server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: added peer, starting replication: peer=server-c01ec779-f38f-c9ba-1968-37440a72b261 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.733Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.628111617 +0000 UTC m=+11.753137788 (1 nodes), 51.060549ms from start of poll, 51.596754ms from cluster start. Timeout at 2021-10-19 21:54:30.733229573 +0000 UTC m=+11.858255724, 105.117936ms after stability testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.733Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: starting snapshot up to: index=12 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.742Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: snapshot complete up to: index=12 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: restored user snapshot: index=1 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.752Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.755Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: Installed remote snapshot testLoggerAdapter verbose: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [WARN] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: skipping application of old log: index=22 testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.756Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testLoggerAdapter verbose: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.757Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.758Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: Installed remote snapshot testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.758Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.772Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testLoggerAdapter verbose: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.772Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.775Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.775Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering follower state: follower="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Follower]" leader= testLoggerAdapter verbose: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.775Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testLoggerAdapter verbose: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.775Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: entering follower state: follower="Node at 0c27add1-31df-77c4-1e72-fc0de87ba883 [Follower]" leader= testLoggerAdapter verbose: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.775Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.775Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.775Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: entering follower state: follower="Node at 0388b01a-0fae-7c71-d88b-885be1f09aef [Follower]" leader= testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.833Z [WARN] server-31b9c99b-a223-3cff-4490-481511dffa8c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.833Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering candidate state: node="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Candidate]" term=2 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: election won: tally=2 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering leader state: leader="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Leader]" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: added peer, starting replication: peer=server-0c27add1-31df-77c4-1e72-fc0de87ba883 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: added peer, starting replication: peer=server-0388b01a-0fae-7c71-d88b-885be1f09aef testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.939Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.834332104 +0000 UTC m=+11.959358244 (1 nodes), 58.372292ms from start of poll, 59.14785ms from cluster start. Timeout at 2021-10-19 21:54:30.939366455 +0000 UTC m=+12.064392600, 105.034356ms after stability testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.940Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: starting snapshot up to: index=12 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.944Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: snapshot complete up to: index=12 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.946Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: restored user snapshot: index=1 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.955Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.955Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: Installed remote snapshot testLoggerAdapter verbose: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [WARN] server-0c27add1-31df-77c4-1e72-fc0de87ba883: skipping application of old log: index=22 testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.958Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testLoggerAdapter verbose: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.958Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.959Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: Installed remote snapshot testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.959Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.974Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testLoggerAdapter verbose: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.974Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testLoggerAdapter verbose: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:30.978Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testLoggerAdapter verbose: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:30.978Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: entering follower state: follower="Node at 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae [Follower]" leader= testLoggerAdapter verbose: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:30.978Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testLoggerAdapter verbose: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:30.978Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: entering follower state: follower="Node at 5228fed4-8c59-d2a3-f2c9-51f3318c20bb [Follower]" leader= testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:30.978Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:30.978Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:30.978Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering follower state: follower="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Follower]" leader= testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.035Z [WARN] server-4e190d97-2893-4874-eec7-868c2b2f38cc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering candidate state: node="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Candidate]" term=2 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: election won: tally=2 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering leader state: leader="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Leader]" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: added peer, starting replication: peer=server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: added peer, starting replication: peer=server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.141Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.036403081 +0000 UTC m=+12.161429283 (1 nodes), 57.656635ms from start of poll, 58.37608ms from cluster start. Timeout at 2021-10-19 21:54:31.141506187 +0000 UTC m=+12.266532323, 105.10304ms after stability testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.142Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: starting snapshot up to: index=12 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.145Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: snapshot complete up to: index=12 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: restored user snapshot: index=1 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.156Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.158Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testLoggerAdapter verbose: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: Installed remote snapshot testLoggerAdapter verbose: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [WARN] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: skipping application of old log: index=22 testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.160Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testLoggerAdapter verbose: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:31.160Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: copied to local snapshot: bytes=71 testLoggerAdapter verbose: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:31.160Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: Installed remote snapshot testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.160Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.176Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testLoggerAdapter verbose: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.176Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" --- PASS: TestRaft_UserRestore (1.26s) testing.go:149: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:29.921Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testing.go:149: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:29.921Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: entering follower state: follower="Node at 469a5b6d-8759-46c6-040e-c8b25b7c2908 [Follower]" leader= testing.go:149: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:29.921Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testing.go:149: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:29.921Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: entering follower state: follower="Node at d92b10e5-d2fd-9860-e66a-1e9725eb1579 [Follower]" leader= testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:29.922Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-469a5b6d-8759-46c6-040e-c8b25b7c2908 Address:469a5b6d-8759-46c6-040e-c8b25b7c2908} {Suffrage:Voter ID:server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 Address:d92b10e5-d2fd-9860-e66a-1e9725eb1579} {Suffrage:Voter ID:server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e Address:7b9320c8-c18a-e748-ba48-3f508a9a3f3e}]" testing.go:149: cluster: 2021-10-19T21:54:29.922Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:29.922Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering follower state: follower="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Follower]" leader= testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [WARN] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering candidate state: node="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Candidate]" term=2 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: election won: tally=2 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: entering leader state: leader="Node at 7b9320c8-c18a-e748-ba48-3f508a9a3f3e [Leader]" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: added peer, starting replication: peer=server-469a5b6d-8759-46c6-040e-c8b25b7c2908 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: added peer, starting replication: peer=server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.000Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testing.go:149: cluster: 2021-10-19T21:54:30.105Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.000650293 +0000 UTC m=+11.125676416 (1 nodes), 78.371947ms from start of poll, 79.702081ms from cluster start. Timeout at 2021-10-19 21:54:30.105659082 +0000 UTC m=+11.230685224, 105.008808ms after stability testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.106Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:30.106Z [INFO] creating new snapshot: path=/tmp/raft204610783/snapshots/2-12-1634680470106.tmp testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.109Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:30.109Z [INFO] creating new snapshot: path=/tmp/raft204610783/snapshots/2-23-1634680470109.tmp testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: copied to local snapshot: bytes=71 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: restored user snapshot: index=1 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.113Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.114Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.120Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.120Z [INFO] creating new snapshot: path=/tmp/raft031862117/snapshots/2-23-1634680470120.tmp testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.120Z [ERROR] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.121Z [INFO] creating new snapshot: path=/tmp/raft318248315/snapshots/2-23-1634680470121.tmp testing.go:149: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.124Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: copied to local snapshot: bytes=71 testing.go:149: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:30.125Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: copied to local snapshot: bytes=71 testing.go:149: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.125Z [INFO] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: Installed remote snapshot testing.go:149: server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: 2021-10-19T21:54:30.125Z [WARN] server-d92b10e5-d2fd-9860-e66a-1e9725eb1579: skipping application of old log: index=22 testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.125Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testing.go:149: server-469a5b6d-8759-46c6-040e-c8b25b7c2908: 2021-10-19T21:54:30.125Z [INFO] server-469a5b6d-8759-46c6-040e-c8b25b7c2908: Installed remote snapshot testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.125Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: pipelining replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.136Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-d92b10e5-d2fd-9860-e66a-1e9725eb1579 d92b10e5-d2fd-9860-e66a-1e9725eb1579}" testing.go:149: server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: 2021-10-19T21:54:30.136Z [INFO] server-7b9320c8-c18a-e748-ba48-3f508a9a3f3e: aborting pipeline replication: peer="{Voter server-469a5b6d-8759-46c6-040e-c8b25b7c2908 469a5b6d-8759-46c6-040e-c8b25b7c2908}" testing.go:149: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.138Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testing.go:149: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.139Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: entering follower state: follower="Node at 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 [Follower]" leader= testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.139Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testing.go:149: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.139Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 Address:41f8a113-ae61-e2d9-ff3c-fb7b41e532b3} {Suffrage:Voter ID:server-9008c103-20bb-b91b-ec57-0716efd507f6 Address:9008c103-20bb-b91b-ec57-0716efd507f6} {Suffrage:Voter ID:server-81228510-69dc-83c7-166d-0d5e5826bac1 Address:81228510-69dc-83c7-166d-0d5e5826bac1}]" testing.go:149: cluster: 2021-10-19T21:54:30.140Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.140Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering follower state: follower="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Follower]" leader= testing.go:149: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.140Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: entering follower state: follower="Node at 81228510-69dc-83c7-166d-0d5e5826bac1 [Follower]" leader= testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [WARN] server-9008c103-20bb-b91b-ec57-0716efd507f6: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering candidate state: node="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Candidate]" term=2 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: election won: tally=2 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: entering leader state: leader="Node at 9008c103-20bb-b91b-ec57-0716efd507f6 [Leader]" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: added peer, starting replication: peer=server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: added peer, starting replication: peer=server-81228510-69dc-83c7-166d-0d5e5826bac1 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.203Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testing.go:149: cluster: 2021-10-19T21:54:30.308Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.203325515 +0000 UTC m=+11.328351656 (1 nodes), 63.269967ms from start of poll, 64.606969ms from cluster start. Timeout at 2021-10-19 21:54:30.30841024 +0000 UTC m=+11.433436400, 105.084744ms after stability testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.308Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:30.308Z [INFO] creating new snapshot: path=/tmp/raft044508291/snapshots/2-12-1634680470308.tmp testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.312Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:30.313Z [INFO] creating new snapshot: path=/tmp/raft044508291/snapshots/2-23-1634680470313.tmp testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.316Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: copied to local snapshot: bytes=71 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: restored user snapshot: index=1 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.317Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.323Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.324Z [INFO] creating new snapshot: path=/tmp/raft664664493/snapshots/2-23-1634680470324.tmp testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.326Z [ERROR] server-9008c103-20bb-b91b-ec57-0716efd507f6: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.326Z [INFO] creating new snapshot: path=/tmp/raft769878633/snapshots/2-23-1634680470326.tmp testing.go:149: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.328Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: copied to local snapshot: bytes=71 testing.go:149: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.329Z [INFO] server-81228510-69dc-83c7-166d-0d5e5826bac1: Installed remote snapshot testing.go:149: server-81228510-69dc-83c7-166d-0d5e5826bac1: 2021-10-19T21:54:30.329Z [WARN] server-81228510-69dc-83c7-166d-0d5e5826bac1: skipping application of old log: index=22 testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.329Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testing.go:149: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.330Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: copied to local snapshot: bytes=71 testing.go:149: server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: 2021-10-19T21:54:30.330Z [INFO] server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3: Installed remote snapshot testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.330Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: pipelining replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.345Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-81228510-69dc-83c7-166d-0d5e5826bac1 81228510-69dc-83c7-166d-0d5e5826bac1}" testing.go:149: server-9008c103-20bb-b91b-ec57-0716efd507f6: 2021-10-19T21:54:30.345Z [INFO] server-9008c103-20bb-b91b-ec57-0716efd507f6: aborting pipeline replication: peer="{Voter server-41f8a113-ae61-e2d9-ff3c-fb7b41e532b3 41f8a113-ae61-e2d9-ff3c-fb7b41e532b3}" testing.go:149: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.348Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testing.go:149: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.348Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: entering follower state: follower="Node at 7477a398-86af-21be-b681-8103cbb497f1 [Follower]" leader= testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.348Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.348Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering follower state: follower="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Follower]" leader= testing.go:149: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.348Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7477a398-86af-21be-b681-8103cbb497f1 Address:7477a398-86af-21be-b681-8103cbb497f1} {Suffrage:Voter ID:server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 Address:90a8d945-e3a3-26b7-d6d6-0e24abd34ea9} {Suffrage:Voter ID:server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 Address:c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}]" testing.go:149: cluster: 2021-10-19T21:54:30.348Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.348Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: entering follower state: follower="Node at c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 [Follower]" leader= testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.428Z [WARN] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering candidate state: node="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Candidate]" term=2 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: election won: tally=2 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: entering leader state: leader="Node at 90a8d945-e3a3-26b7-d6d6-0e24abd34ea9 [Leader]" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: added peer, starting replication: peer=server-7477a398-86af-21be-b681-8103cbb497f1 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: added peer, starting replication: peer=server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.429Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testing.go:149: cluster: 2021-10-19T21:54:30.534Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.429420484 +0000 UTC m=+11.554446672 (1 nodes), 80.492678ms from start of poll, 81.176712ms from cluster start. Timeout at 2021-10-19 21:54:30.534519846 +0000 UTC m=+11.659545996, 105.099324ms after stability testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.535Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:30.535Z [INFO] creating new snapshot: path=/tmp/raft052790065/snapshots/2-12-1634680470535.tmp testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.539Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:30.541Z [INFO] creating new snapshot: path=/tmp/raft052790065/snapshots/2-23-1634680470541.tmp testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: copied to local snapshot: bytes=71 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: restored user snapshot: index=1 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.543Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.549Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.550Z [INFO] creating new snapshot: path=/tmp/raft132467339/snapshots/2-23-1634680470550.tmp testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.551Z [ERROR] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.552Z [INFO] creating new snapshot: path=/tmp/raft665966695/snapshots/2-23-1634680470552.tmp testing.go:149: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: copied to local snapshot: bytes=71 testing.go:149: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [INFO] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: Installed remote snapshot testing.go:149: server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: 2021-10-19T21:54:30.552Z [WARN] server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5: skipping application of old log: index=22 testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.552Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testing.go:149: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.554Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: copied to local snapshot: bytes=71 testing.go:149: server-7477a398-86af-21be-b681-8103cbb497f1: 2021-10-19T21:54:30.554Z [INFO] server-7477a398-86af-21be-b681-8103cbb497f1: Installed remote snapshot testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.554Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: pipelining replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.573Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-c2eef6bc-5e2e-a787-5fd7-e818af16d4f5 c2eef6bc-5e2e-a787-5fd7-e818af16d4f5}" testing.go:149: server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: 2021-10-19T21:54:30.574Z [INFO] server-90a8d945-e3a3-26b7-d6d6-0e24abd34ea9: aborting pipeline replication: peer="{Voter server-7477a398-86af-21be-b681-8103cbb497f1 7477a398-86af-21be-b681-8103cbb497f1}" testing.go:149: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.576Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testing.go:149: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.576Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: entering follower state: follower="Node at 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 [Follower]" leader= testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.576Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.576Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering follower state: follower="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Follower]" leader= testing.go:149: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.576Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 Address:9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4} {Suffrage:Voter ID:server-44e8c40e-37f9-b29c-ba89-fd638f92219d Address:44e8c40e-37f9-b29c-ba89-fd638f92219d} {Suffrage:Voter ID:server-c01ec779-f38f-c9ba-1968-37440a72b261 Address:c01ec779-f38f-c9ba-1968-37440a72b261}]" testing.go:149: cluster: 2021-10-19T21:54:30.577Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.577Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: entering follower state: follower="Node at c01ec779-f38f-c9ba-1968-37440a72b261 [Follower]" leader= testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.627Z [WARN] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.627Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering candidate state: node="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Candidate]" term=2 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: election won: tally=2 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: entering leader state: leader="Node at 44e8c40e-37f9-b29c-ba89-fd638f92219d [Leader]" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: added peer, starting replication: peer=server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: added peer, starting replication: peer=server-c01ec779-f38f-c9ba-1968-37440a72b261 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.628Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testing.go:149: cluster: 2021-10-19T21:54:30.733Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.628111617 +0000 UTC m=+11.753137788 (1 nodes), 51.060549ms from start of poll, 51.596754ms from cluster start. Timeout at 2021-10-19 21:54:30.733229573 +0000 UTC m=+11.858255724, 105.117936ms after stability testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.733Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:30.733Z [INFO] creating new snapshot: path=/tmp/raft297886447/snapshots/2-12-1634680470733.tmp testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.742Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:30.743Z [INFO] creating new snapshot: path=/tmp/raft297886447/snapshots/2-113-1634680470743.tmp testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: copied to local snapshot: bytes=71 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: restored user snapshot: index=1 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.746Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.752Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.753Z [INFO] creating new snapshot: path=/tmp/raft651644661/snapshots/2-113-1634680470753.tmp testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.755Z [ERROR] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.755Z [INFO] creating new snapshot: path=/tmp/raft630900985/snapshots/2-113-1634680470755.tmp testing.go:149: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: copied to local snapshot: bytes=71 testing.go:149: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [INFO] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: Installed remote snapshot testing.go:149: server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: 2021-10-19T21:54:30.756Z [WARN] server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4: skipping application of old log: index=22 testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.756Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testing.go:149: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.757Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: copied to local snapshot: bytes=71 testing.go:149: server-c01ec779-f38f-c9ba-1968-37440a72b261: 2021-10-19T21:54:30.758Z [INFO] server-c01ec779-f38f-c9ba-1968-37440a72b261: Installed remote snapshot testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.758Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: pipelining replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.772Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-c01ec779-f38f-c9ba-1968-37440a72b261 c01ec779-f38f-c9ba-1968-37440a72b261}" testing.go:149: server-44e8c40e-37f9-b29c-ba89-fd638f92219d: 2021-10-19T21:54:30.772Z [INFO] server-44e8c40e-37f9-b29c-ba89-fd638f92219d: aborting pipeline replication: peer="{Voter server-9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4 9ab6f498-0d5f-fd38-b3f8-91d662e7a3a4}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.775Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.775Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering follower state: follower="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Follower]" leader= testing.go:149: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.775Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testing.go:149: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.775Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: entering follower state: follower="Node at 0c27add1-31df-77c4-1e72-fc0de87ba883 [Follower]" leader= testing.go:149: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.775Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-31b9c99b-a223-3cff-4490-481511dffa8c Address:31b9c99b-a223-3cff-4490-481511dffa8c} {Suffrage:Voter ID:server-0c27add1-31df-77c4-1e72-fc0de87ba883 Address:0c27add1-31df-77c4-1e72-fc0de87ba883} {Suffrage:Voter ID:server-0388b01a-0fae-7c71-d88b-885be1f09aef Address:0388b01a-0fae-7c71-d88b-885be1f09aef}]" testing.go:149: cluster: 2021-10-19T21:54:30.775Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.775Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: entering follower state: follower="Node at 0388b01a-0fae-7c71-d88b-885be1f09aef [Follower]" leader= testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.833Z [WARN] server-31b9c99b-a223-3cff-4490-481511dffa8c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.833Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering candidate state: node="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Candidate]" term=2 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: election won: tally=2 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: entering leader state: leader="Node at 31b9c99b-a223-3cff-4490-481511dffa8c [Leader]" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: added peer, starting replication: peer=server-0c27add1-31df-77c4-1e72-fc0de87ba883 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: added peer, starting replication: peer=server-0388b01a-0fae-7c71-d88b-885be1f09aef testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.834Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testing.go:149: cluster: 2021-10-19T21:54:30.939Z [INFO] stable state for Leader reached at 2021-10-19 21:54:30.834332104 +0000 UTC m=+11.959358244 (1 nodes), 58.372292ms from start of poll, 59.14785ms from cluster start. Timeout at 2021-10-19 21:54:30.939366455 +0000 UTC m=+12.064392600, 105.034356ms after stability testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.940Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:30.940Z [INFO] creating new snapshot: path=/tmp/raft673463187/snapshots/2-12-1634680470940.tmp testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.944Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:30.944Z [INFO] creating new snapshot: path=/tmp/raft673463187/snapshots/2-1013-1634680470944.tmp testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.946Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: copied to local snapshot: bytes=71 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: restored user snapshot: index=1 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.947Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.955Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.955Z [ERROR] server-31b9c99b-a223-3cff-4490-481511dffa8c: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:30.955Z [INFO] creating new snapshot: path=/tmp/raft865315645/snapshots/2-1013-1634680470955.tmp testing.go:153: 2021-10-19T21:54:30.955Z [INFO] creating new snapshot: path=/tmp/raft437474935/snapshots/2-1013-1634680470955.tmp testing.go:149: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: copied to local snapshot: bytes=71 testing.go:149: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [INFO] server-0c27add1-31df-77c4-1e72-fc0de87ba883: Installed remote snapshot testing.go:149: server-0c27add1-31df-77c4-1e72-fc0de87ba883: 2021-10-19T21:54:30.958Z [WARN] server-0c27add1-31df-77c4-1e72-fc0de87ba883: skipping application of old log: index=22 testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.958Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testing.go:149: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.958Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: copied to local snapshot: bytes=71 testing.go:149: server-0388b01a-0fae-7c71-d88b-885be1f09aef: 2021-10-19T21:54:30.959Z [INFO] server-0388b01a-0fae-7c71-d88b-885be1f09aef: Installed remote snapshot testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.959Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: pipelining replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.974Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0388b01a-0fae-7c71-d88b-885be1f09aef 0388b01a-0fae-7c71-d88b-885be1f09aef}" testing.go:149: server-31b9c99b-a223-3cff-4490-481511dffa8c: 2021-10-19T21:54:30.974Z [INFO] server-31b9c99b-a223-3cff-4490-481511dffa8c: aborting pipeline replication: peer="{Voter server-0c27add1-31df-77c4-1e72-fc0de87ba883 0c27add1-31df-77c4-1e72-fc0de87ba883}" testing.go:149: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:30.978Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testing.go:149: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:30.978Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: entering follower state: follower="Node at 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae [Follower]" leader= testing.go:149: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:30.978Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testing.go:149: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:30.978Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: entering follower state: follower="Node at 5228fed4-8c59-d2a3-f2c9-51f3318c20bb [Follower]" leader= testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:30.978Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae Address:0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae} {Suffrage:Voter ID:server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb Address:5228fed4-8c59-d2a3-f2c9-51f3318c20bb} {Suffrage:Voter ID:server-4e190d97-2893-4874-eec7-868c2b2f38cc Address:4e190d97-2893-4874-eec7-868c2b2f38cc}]" testing.go:149: cluster: 2021-10-19T21:54:30.978Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:30.978Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering follower state: follower="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Follower]" leader= testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.035Z [WARN] server-4e190d97-2893-4874-eec7-868c2b2f38cc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering candidate state: node="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Candidate]" term=2 testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: election won: tally=2 testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: entering leader state: leader="Node at 4e190d97-2893-4874-eec7-868c2b2f38cc [Leader]" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: added peer, starting replication: peer=server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: added peer, starting replication: peer=server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.036Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testing.go:149: cluster: 2021-10-19T21:54:31.141Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.036403081 +0000 UTC m=+12.161429283 (1 nodes), 57.656635ms from start of poll, 58.37608ms from cluster start. Timeout at 2021-10-19 21:54:31.141506187 +0000 UTC m=+12.266532323, 105.10304ms after stability testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.142Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: starting snapshot up to: index=12 testing.go:153: 2021-10-19T21:54:31.142Z [INFO] creating new snapshot: path=/tmp/raft944889477/snapshots/2-12-1634680471142.tmp testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.145Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: snapshot complete up to: index=12 testing.go:153: 2021-10-19T21:54:31.146Z [INFO] creating new snapshot: path=/tmp/raft944889477/snapshots/2-10013-1634680471146.tmp testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: copied to local snapshot: bytes=71 testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: restored user snapshot: index=1 testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.148Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.156Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:31.157Z [INFO] creating new snapshot: path=/tmp/raft643021211/snapshots/2-10013-1634680471157.tmp testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.158Z [ERROR] server-4e190d97-2893-4874-eec7-868c2b2f38cc: failed to get log: index=23 error="log not found" testing.go:153: 2021-10-19T21:54:31.158Z [INFO] creating new snapshot: path=/tmp/raft905337793/snapshots/2-10013-1634680471158.tmp testing.go:149: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: copied to local snapshot: bytes=71 testing.go:149: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [INFO] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: Installed remote snapshot testing.go:149: server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: 2021-10-19T21:54:31.160Z [WARN] server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb: skipping application of old log: index=22 testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.160Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testing.go:149: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:31.160Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: copied to local snapshot: bytes=71 testing.go:149: server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: 2021-10-19T21:54:31.160Z [INFO] server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae: Installed remote snapshot testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.160Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: pipelining replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.176Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-5228fed4-8c59-d2a3-f2c9-51f3318c20bb 5228fed4-8c59-d2a3-f2c9-51f3318c20bb}" testing.go:149: server-4e190d97-2893-4874-eec7-868c2b2f38cc: 2021-10-19T21:54:31.176Z [INFO] server-4e190d97-2893-4874-eec7-868c2b2f38cc: aborting pipeline replication: peer="{Voter server-0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae 0f717edf-c4ed-7c6a-1a7d-73f3ca7ac1ae}" === RUN TestRaft_SendSnapshotFollower testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.179Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.179Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: entering follower state: follower="Node at db4f6d2d-49db-a99f-6164-826e8779288a [Follower]" leader= testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.179Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.179Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering follower state: follower="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Follower]" leader= testLoggerAdapter verbose: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.179Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.179Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.179Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: entering follower state: follower="Node at 790662ad-38ca-19b3-facd-7a50068e40e3 [Follower]" leader= testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [WARN] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering candidate state: node="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Candidate]" term=2 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: election won: tally=2 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering leader state: leader="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Leader]" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: added peer, starting replication: peer=server-db4f6d2d-49db-a99f-6164-826e8779288a testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: added peer, starting replication: peer=server-790662ad-38ca-19b3-facd-7a50068e40e3 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-790662ad-38ca-19b3-facd-7a50068e40e3 790662ad-38ca-19b3-facd-7a50068e40e3}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.339Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.234124779 +0000 UTC m=+12.359151001 (2 nodes), 54.410251ms from start of poll, 55.107309ms from cluster start. Timeout at 2021-10-19 21:54:31.339207263 +0000 UTC m=+12.464233400, 105.082399ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.339Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.444Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.339456729 +0000 UTC m=+12.464482857 (1 nodes), 1.95µs from start of poll, 160.439165ms from cluster start. Timeout at 2021-10-19 21:54:31.444537485 +0000 UTC m=+12.569563675, 105.080818ms after stability testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to pipeline appendEntries: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" error="append pipeline closed" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to appendEntries to: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" error="failed to connect to peer: db4f6d2d-49db-a99f-6164-826e8779288a" testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.449Z [ERROR] server-db4f6d2d-49db-a99f-6164-826e8779288a: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.449Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: starting snapshot up to: index=102 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.450Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to heartbeat to: peer=db4f6d2d-49db-a99f-6164-826e8779288a error="failed to connect to peer: db4f6d2d-49db-a99f-6164-826e8779288a" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.452Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.453Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: snapshot complete up to: index=102 testLoggerAdapter verbose: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.453Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: starting snapshot up to: index=98 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.455Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.455Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.455Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: snapshot complete up to: index=98 testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.457Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.458Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: Installed remote snapshot testLoggerAdapter verbose: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.458Z [WARN] server-db4f6d2d-49db-a99f-6164-826e8779288a: skipping application of old log: index=102 testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.458Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.460Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-790662ad-38ca-19b3-facd-7a50068e40e3 790662ad-38ca-19b3-facd-7a50068e40e3}" testLoggerAdapter verbose: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.460Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" --- PASS: TestRaft_SendSnapshotFollower (0.28s) testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.179Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.179Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: entering follower state: follower="Node at db4f6d2d-49db-a99f-6164-826e8779288a [Follower]" leader= testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.179Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.179Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering follower state: follower="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Follower]" leader= testing.go:149: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.179Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-db4f6d2d-49db-a99f-6164-826e8779288a Address:db4f6d2d-49db-a99f-6164-826e8779288a} {Suffrage:Voter ID:server-ad16b977-7724-6360-a3f8-efd0d733ce6a Address:ad16b977-7724-6360-a3f8-efd0d733ce6a} {Suffrage:Voter ID:server-790662ad-38ca-19b3-facd-7a50068e40e3 Address:790662ad-38ca-19b3-facd-7a50068e40e3}]" testing.go:149: cluster: 2021-10-19T21:54:31.179Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.179Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: entering follower state: follower="Node at 790662ad-38ca-19b3-facd-7a50068e40e3 [Follower]" leader= testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [WARN] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering candidate state: node="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Candidate]" term=2 testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.233Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: election won: tally=2 testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: entering leader state: leader="Node at ad16b977-7724-6360-a3f8-efd0d733ce6a [Leader]" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: added peer, starting replication: peer=server-db4f6d2d-49db-a99f-6164-826e8779288a testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: added peer, starting replication: peer=server-790662ad-38ca-19b3-facd-7a50068e40e3 testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.234Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-790662ad-38ca-19b3-facd-7a50068e40e3 790662ad-38ca-19b3-facd-7a50068e40e3}" testing.go:149: cluster: 2021-10-19T21:54:31.339Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.234124779 +0000 UTC m=+12.359151001 (2 nodes), 54.410251ms from start of poll, 55.107309ms from cluster start. Timeout at 2021-10-19 21:54:31.339207263 +0000 UTC m=+12.464233400, 105.082399ms after stability testing.go:149: cluster: 2021-10-19T21:54:31.339Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:31.444Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.339456729 +0000 UTC m=+12.464482857 (1 nodes), 1.95µs from start of poll, 160.439165ms from cluster start. Timeout at 2021-10-19 21:54:31.444537485 +0000 UTC m=+12.569563675, 105.080818ms after stability testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to pipeline appendEntries: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" error="append pipeline closed" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.444Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to appendEntries to: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" error="failed to connect to peer: db4f6d2d-49db-a99f-6164-826e8779288a" raft_test.go:1320: [INFO] Finished apply without behind follower testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.449Z [ERROR] server-db4f6d2d-49db-a99f-6164-826e8779288a: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.449Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: starting snapshot up to: index=102 testing.go:153: 2021-10-19T21:54:31.449Z [INFO] creating new snapshot: path=/tmp/raft484717449/snapshots/2-102-1634680471449.tmp testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.450Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to heartbeat to: peer=db4f6d2d-49db-a99f-6164-826e8779288a error="failed to connect to peer: db4f6d2d-49db-a99f-6164-826e8779288a" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.452Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: compacting logs: from=1 to=92 testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.453Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: snapshot complete up to: index=102 testing.go:149: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.453Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: starting snapshot up to: index=98 testing.go:153: 2021-10-19T21:54:31.453Z [INFO] creating new snapshot: path=/tmp/raft050872995/snapshots/2-98-1634680471453.tmp testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.455Z [ERROR] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: failed to get log: index=2 error="log not found" testing.go:149: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.455Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: compacting logs: from=1 to=92 testing.go:149: server-790662ad-38ca-19b3-facd-7a50068e40e3: 2021-10-19T21:54:31.455Z [INFO] server-790662ad-38ca-19b3-facd-7a50068e40e3: snapshot complete up to: index=98 testing.go:153: 2021-10-19T21:54:31.455Z [INFO] creating new snapshot: path=/tmp/raft573190399/snapshots/2-102-1634680471455.tmp testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.457Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: copied to local snapshot: bytes=693 testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.458Z [INFO] server-db4f6d2d-49db-a99f-6164-826e8779288a: Installed remote snapshot testing.go:149: server-db4f6d2d-49db-a99f-6164-826e8779288a: 2021-10-19T21:54:31.458Z [WARN] server-db4f6d2d-49db-a99f-6164-826e8779288a: skipping application of old log: index=102 testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.458Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: pipelining replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.460Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-790662ad-38ca-19b3-facd-7a50068e40e3 790662ad-38ca-19b3-facd-7a50068e40e3}" testing.go:149: server-ad16b977-7724-6360-a3f8-efd0d733ce6a: 2021-10-19T21:54:31.460Z [INFO] server-ad16b977-7724-6360-a3f8-efd0d733ce6a: aborting pipeline replication: peer="{Voter server-db4f6d2d-49db-a99f-6164-826e8779288a db4f6d2d-49db-a99f-6164-826e8779288a}" === RUN TestRaft_SendSnapshotAndLogsFollower testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.463Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.463Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering follower state: follower="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Follower]" leader= testLoggerAdapter verbose: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.463Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testLoggerAdapter verbose: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.464Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: entering follower state: follower="Node at 69091216-5dbb-e3cf-ac7d-31ae23de9576 [Follower]" leader= testLoggerAdapter verbose: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.464Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.464Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.464Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: entering follower state: follower="Node at d9175926-3b91-a7be-0991-fbf7972c406e [Follower]" leader= testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [WARN] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering candidate state: node="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Candidate]" term=2 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: election won: tally=2 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering leader state: leader="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Leader]" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: added peer, starting replication: peer=server-69091216-5dbb-e3cf-ac7d-31ae23de9576 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: added peer, starting replication: peer=server-d9175926-3b91-a7be-0991-fbf7972c406e testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-d9175926-3b91-a7be-0991-fbf7972c406e d9175926-3b91-a7be-0991-fbf7972c406e}" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.627Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.522575739 +0000 UTC m=+12.647601865 (2 nodes), 58.29883ms from start of poll, 59.10707ms from cluster start. Timeout at 2021-10-19 21:54:31.627649663 +0000 UTC m=+12.752675854, 105.073989ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.627Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.733Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.627949943 +0000 UTC m=+12.752976117 (1 nodes), 1.5µs from start of poll, 164.481322ms from cluster start. Timeout at 2021-10-19 21:54:31.733039826 +0000 UTC m=+12.858065987, 105.08987ms after stability testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to pipeline appendEntries: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="append pipeline closed" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to appendEntries to: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.734Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to heartbeat to: peer=69091216-5dbb-e3cf-ac7d-31ae23de9576 error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.736Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: starting snapshot up to: index=102 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.741Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.742Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: snapshot complete up to: index=102 testLoggerAdapter verbose: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.742Z [ERROR] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: failed to take snapshot: error="nothing new to snapshot" testLoggerAdapter verbose: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.742Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: starting snapshot up to: index=98 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.743Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.744Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to install snapshot: id=2-102-1634680471736 error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.744Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to send snapshot to: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testLoggerAdapter verbose: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.747Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: compacting logs: from=1 to=92 testLoggerAdapter verbose: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.747Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: snapshot complete up to: index=98 testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.754Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to get log: index=2 error="log not found" testLoggerAdapter verbose: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.758Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: copied to local snapshot: bytes=693 testLoggerAdapter verbose: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.758Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: Installed remote snapshot testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.758Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.762Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testLoggerAdapter verbose: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.762Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-d9175926-3b91-a7be-0991-fbf7972c406e d9175926-3b91-a7be-0991-fbf7972c406e}" --- PASS: TestRaft_SendSnapshotAndLogsFollower (0.30s) testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.463Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.463Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering follower state: follower="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Follower]" leader= testing.go:149: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.463Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testing.go:149: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.464Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: entering follower state: follower="Node at 69091216-5dbb-e3cf-ac7d-31ae23de9576 [Follower]" leader= testing.go:149: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.464Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa Address:e6b6837d-48a5-aae3-e97c-8f4e1664d6aa} {Suffrage:Voter ID:server-69091216-5dbb-e3cf-ac7d-31ae23de9576 Address:69091216-5dbb-e3cf-ac7d-31ae23de9576} {Suffrage:Voter ID:server-d9175926-3b91-a7be-0991-fbf7972c406e Address:d9175926-3b91-a7be-0991-fbf7972c406e}]" testing.go:149: cluster: 2021-10-19T21:54:31.464Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.464Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: entering follower state: follower="Node at d9175926-3b91-a7be-0991-fbf7972c406e [Follower]" leader= testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [WARN] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering candidate state: node="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Candidate]" term=2 testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: election won: tally=2 testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: entering leader state: leader="Node at e6b6837d-48a5-aae3-e97c-8f4e1664d6aa [Leader]" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: added peer, starting replication: peer=server-69091216-5dbb-e3cf-ac7d-31ae23de9576 testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: added peer, starting replication: peer=server-d9175926-3b91-a7be-0991-fbf7972c406e testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-d9175926-3b91-a7be-0991-fbf7972c406e d9175926-3b91-a7be-0991-fbf7972c406e}" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.522Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testing.go:149: cluster: 2021-10-19T21:54:31.627Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.522575739 +0000 UTC m=+12.647601865 (2 nodes), 58.29883ms from start of poll, 59.10707ms from cluster start. Timeout at 2021-10-19 21:54:31.627649663 +0000 UTC m=+12.752675854, 105.073989ms after stability testing.go:149: cluster: 2021-10-19T21:54:31.627Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:31.733Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.627949943 +0000 UTC m=+12.752976117 (1 nodes), 1.5µs from start of poll, 164.481322ms from cluster start. Timeout at 2021-10-19 21:54:31.733039826 +0000 UTC m=+12.858065987, 105.08987ms after stability testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to pipeline appendEntries: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="append pipeline closed" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.733Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to appendEntries to: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.734Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to heartbeat to: peer=69091216-5dbb-e3cf-ac7d-31ae23de9576 error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" raft_test.go:1362: [INFO] Finished apply without behind follower testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.736Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: starting snapshot up to: index=102 testing.go:153: 2021-10-19T21:54:31.736Z [INFO] creating new snapshot: path=/tmp/raft528229069/snapshots/2-102-1634680471736.tmp testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.741Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: compacting logs: from=1 to=92 testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.742Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: snapshot complete up to: index=102 testing.go:149: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.742Z [ERROR] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: failed to take snapshot: error="nothing new to snapshot" testing.go:149: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.742Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: starting snapshot up to: index=98 testing.go:153: 2021-10-19T21:54:31.742Z [INFO] creating new snapshot: path=/tmp/raft731186257/snapshots/2-98-1634680471742.tmp testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.743Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to get log: index=2 error="log not found" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.744Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to install snapshot: id=2-102-1634680471736 error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.744Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to send snapshot to: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" error="failed to connect to peer: 69091216-5dbb-e3cf-ac7d-31ae23de9576" testing.go:149: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.747Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: compacting logs: from=1 to=92 testing.go:149: server-d9175926-3b91-a7be-0991-fbf7972c406e: 2021-10-19T21:54:31.747Z [INFO] server-d9175926-3b91-a7be-0991-fbf7972c406e: snapshot complete up to: index=98 raft_test.go:1383: [INFO] Finished apply without behind follower testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.754Z [ERROR] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: failed to get log: index=2 error="log not found" testing.go:153: 2021-10-19T21:54:31.755Z [INFO] creating new snapshot: path=/tmp/raft030175367/snapshots/2-102-1634680471755.tmp testing.go:149: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.758Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: copied to local snapshot: bytes=693 testing.go:149: server-69091216-5dbb-e3cf-ac7d-31ae23de9576: 2021-10-19T21:54:31.758Z [INFO] server-69091216-5dbb-e3cf-ac7d-31ae23de9576: Installed remote snapshot testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.758Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: pipelining replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.762Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-69091216-5dbb-e3cf-ac7d-31ae23de9576 69091216-5dbb-e3cf-ac7d-31ae23de9576}" testing.go:149: server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: 2021-10-19T21:54:31.762Z [INFO] server-e6b6837d-48a5-aae3-e97c-8f4e1664d6aa: aborting pipeline replication: peer="{Voter server-d9175926-3b91-a7be-0991-fbf7972c406e d9175926-3b91-a7be-0991-fbf7972c406e}" === RUN TestRaft_ReJoinFollower testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:31.765Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:31.765Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering follower state: follower="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Follower]" leader= testLoggerAdapter verbose: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:31.765Z [INFO] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testLoggerAdapter verbose: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:31.765Z [INFO] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: entering follower state: follower="Node at 7a4380c8-93db-2b8c-3e3f-82af696162ce [Follower]" leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.765Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.765Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.765Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering follower state: follower="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Follower]" leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering candidate state: node="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Candidate]" term=2 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: election won: tally=2 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering leader state: leader="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Leader]" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-7a4380c8-93db-2b8c-3e3f-82af696162ce testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.942Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.837617952 +0000 UTC m=+12.962644134 (1 nodes), 72.048157ms from start of poll, 72.700364ms from cluster start. Timeout at 2021-10-19 21:54:31.942650327 +0000 UTC m=+13.067676457, 105.032323ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:31.947Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.053Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.948087698 +0000 UTC m=+13.073113846 (2 nodes), 1.821µs from start of poll, 183.170076ms from cluster start. Timeout at 2021-10-19 21:54:32.053163253 +0000 UTC m=+13.178189391, 105.075545ms after stability testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: updating configuration: command=RemoveServer server-id=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 server-addr= servers="[{Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: removed peer, stopping replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 last-index=3 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.160Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: heartbeat timeout reached, starting election: last-leader=71a93341-9180-680b-0082-9a9efe24ab2c testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.160Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering candidate state: node="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Candidate]" term=3 testLoggerAdapter verbose: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:32.160Z [WARN] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: updating configuration: command=AddStaging server-id=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 server-addr=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 servers="[{Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c} {Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}]" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [ERROR] server-71a93341-9180-680b-0082-9a9efe24ab2c: peer has newer term, stopping replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.160Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering follower state: follower="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Follower]" leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.222Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: Election timeout reached, restarting election testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.222Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering candidate state: node="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Candidate]" term=4 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.223Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: rejecting vote request since our last index is greater: candidate=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 last-index=4 last-candidate-index=3 testLoggerAdapter verbose: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:32.223Z [WARN] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering candidate state: node="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Candidate]" term=5 testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.247Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering follower state: follower="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Follower]" leader= testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: election won: tally=2 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering leader state: leader="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Leader]" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-7a4380c8-93db-2b8c-3e3f-82af696162ce testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testLoggerAdapter verbose: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.247Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: failed to get previous log: previous-index=4 last-index=3 error="log not found" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: appendEntries rejected, sending older logs: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" next=4 testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.352Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.24773617 +0000 UTC m=+13.372762340 (1 nodes), 86.809066ms from start of poll, 482.81857ms from cluster start. Timeout at 2021-10-19 21:54:32.352797647 +0000 UTC m=+13.477823794, 105.061454ms after stability testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.353Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testLoggerAdapter verbose: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.353Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" --- PASS: TestRaft_ReJoinFollower (0.59s) testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:31.765Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:31.765Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering follower state: follower="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Follower]" leader= testing.go:149: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:31.765Z [INFO] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testing.go:149: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:31.765Z [INFO] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: entering follower state: follower="Node at 7a4380c8-93db-2b8c-3e3f-82af696162ce [Follower]" leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.765Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8} {Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testing.go:149: cluster: 2021-10-19T21:54:31.765Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.765Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering follower state: follower="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Follower]" leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering candidate state: node="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Candidate]" term=2 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: election won: tally=2 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering leader state: leader="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Leader]" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.836Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-7a4380c8-93db-2b8c-3e3f-82af696162ce testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:31.837Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testing.go:149: cluster: 2021-10-19T21:54:31.942Z [INFO] stable state for Leader reached at 2021-10-19 21:54:31.837617952 +0000 UTC m=+12.962644134 (1 nodes), 72.048157ms from start of poll, 72.700364ms from cluster start. Timeout at 2021-10-19 21:54:31.942650327 +0000 UTC m=+13.067676457, 105.032323ms after stability testing.go:149: cluster: 2021-10-19T21:54:31.947Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:32.053Z [INFO] stable state for Follower reached at 2021-10-19 21:54:31.948087698 +0000 UTC m=+13.073113846 (2 nodes), 1.821µs from start of poll, 183.170076ms from cluster start. Timeout at 2021-10-19 21:54:32.053163253 +0000 UTC m=+13.178189391, 105.075545ms after stability testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: updating configuration: command=RemoveServer server-id=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 server-addr= servers="[{Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c}]" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: removed peer, stopping replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 last-index=3 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.053Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.160Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: heartbeat timeout reached, starting election: last-leader=71a93341-9180-680b-0082-9a9efe24ab2c testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.160Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering candidate state: node="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Candidate]" term=3 testing.go:149: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:32.160Z [WARN] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: updating configuration: command=AddStaging server-id=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 server-addr=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 servers="[{Suffrage:Voter ID:server-7a4380c8-93db-2b8c-3e3f-82af696162ce Address:7a4380c8-93db-2b8c-3e3f-82af696162ce} {Suffrage:Voter ID:server-71a93341-9180-680b-0082-9a9efe24ab2c Address:71a93341-9180-680b-0082-9a9efe24ab2c} {Suffrage:Voter ID:server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 Address:c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}]" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [ERROR] server-71a93341-9180-680b-0082-9a9efe24ab2c: peer has newer term, stopping replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testing.go:149: cluster: 2021-10-19T21:54:32.160Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering follower state: follower="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Follower]" leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.160Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.222Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: Election timeout reached, restarting election testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.222Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering candidate state: node="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Candidate]" term=4 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.223Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: rejecting vote request since our last index is greater: candidate=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 last-index=4 last-candidate-index=3 testing.go:149: server-7a4380c8-93db-2b8c-3e3f-82af696162ce: 2021-10-19T21:54:32.223Z [WARN] server-7a4380c8-93db-2b8c-3e3f-82af696162ce: rejecting vote request since we have a leader: from=c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 leader=71a93341-9180-680b-0082-9a9efe24ab2c testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering candidate state: node="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Candidate]" term=5 testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.247Z [INFO] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: entering follower state: follower="Node at c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 [Follower]" leader= testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: election won: tally=2 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: entering leader state: leader="Node at 71a93341-9180-680b-0082-9a9efe24ab2c [Leader]" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-7a4380c8-93db-2b8c-3e3f-82af696162ce testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: added peer, starting replication: peer=server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 testing.go:149: server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: 2021-10-19T21:54:32.247Z [WARN] server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8: failed to get previous log: previous-index=4 last-index=3 error="log not found" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [WARN] server-71a93341-9180-680b-0082-9a9efe24ab2c: appendEntries rejected, sending older logs: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" next=4 testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.247Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: pipelining replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" testing.go:149: cluster: 2021-10-19T21:54:32.352Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.24773617 +0000 UTC m=+13.372762340 (1 nodes), 86.809066ms from start of poll, 482.81857ms from cluster start. Timeout at 2021-10-19 21:54:32.352797647 +0000 UTC m=+13.477823794, 105.061454ms after stability testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.353Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-7a4380c8-93db-2b8c-3e3f-82af696162ce 7a4380c8-93db-2b8c-3e3f-82af696162ce}" testing.go:149: server-71a93341-9180-680b-0082-9a9efe24ab2c: 2021-10-19T21:54:32.353Z [INFO] server-71a93341-9180-680b-0082-9a9efe24ab2c: aborting pipeline replication: peer="{Voter server-c23c0a74-e8c4-d433-1544-0c33a7dd7eb8 c23c0a74-e8c4-d433-1544-0c33a7dd7eb8}" === RUN TestRaft_LeaderLeaseExpire testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.354Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e835e-1c4b-a495-fbbe-e072277acadd Address:eb5e835e-1c4b-a495-fbbe-e072277acadd} {Suffrage:Voter ID:server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 Address:bb52d4a2-aeca-3231-8ff9-cc80970cd269}]" testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.354Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering follower state: follower="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Follower]" leader= testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.355Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e835e-1c4b-a495-fbbe-e072277acadd Address:eb5e835e-1c4b-a495-fbbe-e072277acadd} {Suffrage:Voter ID:server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 Address:bb52d4a2-aeca-3231-8ff9-cc80970cd269}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.355Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.355Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering follower state: follower="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Follower]" leader= testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.409Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.409Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=2 testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: election won: tally=2 testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering leader state: leader="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Leader]" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: added peer, starting replication: peer=server-eb5e835e-1c4b-a495-fbbe-e072277acadd testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: pipelining replication: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.515Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.410216809 +0000 UTC m=+13.535243068 (1 nodes), 55.123252ms from start of poll, 55.556564ms from cluster start. Timeout at 2021-10-19 21:54:32.515323946 +0000 UTC m=+13.640350140, 105.107072ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.520Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.626Z [INFO] stable state for Follower reached at 2021-10-19 21:54:32.520814334 +0000 UTC m=+13.645840459 (1 nodes), 1.028µs from start of poll, 166.153955ms from cluster start. Timeout at 2021-10-19 21:54:32.625920946 +0000 UTC m=+13.750947157, 105.106698ms after stability testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.630Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to pipeline appendEntries: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="append pipeline closed" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.630Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: aborting pipeline replication: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.631Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.637Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.648Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.657Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.666Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to contact: server-id=server-eb5e835e-1c4b-a495-fbbe-e072277acadd time=50.12371ms testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to contact quorum of nodes, stepping down testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering follower state: follower="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.673Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: heartbeat timeout reached, starting election: last-leader=bb52d4a2-aeca-3231-8ff9-cc80970cd269 testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=3 testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.674Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: Election timeout reached, restarting election testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=4 testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=3 testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to make requestVote RPC: target="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.778Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.673861357 +0000 UTC m=+13.798887510 (0 nodes), 125.178µs from start of poll, 319.201006ms from cluster start. Timeout at 2021-10-19 21:54:32.778931269 +0000 UTC m=+13.903957408, 105.069898ms after stability testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: Election timeout reached, restarting election testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=4 testLoggerAdapter verbose: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to make requestVote RPC: target="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.853Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: Election timeout reached, restarting election testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.853Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=5 testLoggerAdapter verbose: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.854Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" --- PASS: TestRaft_LeaderLeaseExpire (0.53s) testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.354Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e835e-1c4b-a495-fbbe-e072277acadd Address:eb5e835e-1c4b-a495-fbbe-e072277acadd} {Suffrage:Voter ID:server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 Address:bb52d4a2-aeca-3231-8ff9-cc80970cd269}]" testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.354Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering follower state: follower="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Follower]" leader= testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.355Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-eb5e835e-1c4b-a495-fbbe-e072277acadd Address:eb5e835e-1c4b-a495-fbbe-e072277acadd} {Suffrage:Voter ID:server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 Address:bb52d4a2-aeca-3231-8ff9-cc80970cd269}]" testing.go:149: cluster: 2021-10-19T21:54:32.355Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.355Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering follower state: follower="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Follower]" leader= testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.409Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.409Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=2 testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: election won: tally=2 testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering leader state: leader="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Leader]" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: added peer, starting replication: peer=server-eb5e835e-1c4b-a495-fbbe-e072277acadd testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.410Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: pipelining replication: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" testing.go:149: cluster: 2021-10-19T21:54:32.515Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.410216809 +0000 UTC m=+13.535243068 (1 nodes), 55.123252ms from start of poll, 55.556564ms from cluster start. Timeout at 2021-10-19 21:54:32.515323946 +0000 UTC m=+13.640350140, 105.107072ms after stability testing.go:149: cluster: 2021-10-19T21:54:32.520Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:32.626Z [INFO] stable state for Follower reached at 2021-10-19 21:54:32.520814334 +0000 UTC m=+13.645840459 (1 nodes), 1.028µs from start of poll, 166.153955ms from cluster start. Timeout at 2021-10-19 21:54:32.625920946 +0000 UTC m=+13.750947157, 105.106698ms after stability raft_test.go:1492: [INFO] Disconnecting Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Follower] testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.630Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to pipeline appendEntries: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="append pipeline closed" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.630Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: aborting pipeline replication: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.631Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.637Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.648Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.657Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.666Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to heartbeat to: peer=eb5e835e-1c4b-a495-fbbe-e072277acadd error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to contact: server-id=server-eb5e835e-1c4b-a495-fbbe-e072277acadd time=50.12371ms testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to contact quorum of nodes, stepping down testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.673Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering follower state: follower="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Follower]" leader= testing.go:149: cluster: 2021-10-19T21:54:32.673Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: heartbeat timeout reached, starting election: last-leader=bb52d4a2-aeca-3231-8ff9-cc80970cd269 testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=3 testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.673Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.674Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to appendEntries to: peer="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: Election timeout reached, restarting election testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=4 testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.764Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=3 testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.769Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to make requestVote RPC: target="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: cluster: 2021-10-19T21:54:32.778Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.673861357 +0000 UTC m=+13.798887510 (0 nodes), 125.178µs from start of poll, 319.201006ms from cluster start. Timeout at 2021-10-19 21:54:32.778931269 +0000 UTC m=+13.903957408, 105.069898ms after stability testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [WARN] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: Election timeout reached, restarting election testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [INFO] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: entering candidate state: node="Node at bb52d4a2-aeca-3231-8ff9-cc80970cd269 [Candidate]" term=4 testing.go:149: server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: 2021-10-19T21:54:32.843Z [ERROR] server-bb52d4a2-aeca-3231-8ff9-cc80970cd269: failed to make requestVote RPC: target="{Voter server-eb5e835e-1c4b-a495-fbbe-e072277acadd eb5e835e-1c4b-a495-fbbe-e072277acadd}" error="failed to connect to peer: eb5e835e-1c4b-a495-fbbe-e072277acadd" testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.853Z [WARN] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: Election timeout reached, restarting election testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.853Z [INFO] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: entering candidate state: node="Node at eb5e835e-1c4b-a495-fbbe-e072277acadd [Candidate]" term=5 testing.go:149: server-eb5e835e-1c4b-a495-fbbe-e072277acadd: 2021-10-19T21:54:32.854Z [ERROR] server-eb5e835e-1c4b-a495-fbbe-e072277acadd: failed to make requestVote RPC: target="{Voter server-bb52d4a2-aeca-3231-8ff9-cc80970cd269 bb52d4a2-aeca-3231-8ff9-cc80970cd269}" error="failed to connect to peer: bb52d4a2-aeca-3231-8ff9-cc80970cd269" === RUN TestRaft_Barrier testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.886Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.886Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering follower state: follower="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Follower]" leader= testLoggerAdapter verbose: server-fb20e901-e794-e063-0997-a13fe9bd70f0: 2021-10-19T21:54:32.886Z [INFO] server-fb20e901-e794-e063-0997-a13fe9bd70f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testLoggerAdapter verbose: server-fb20e901-e794-e063-0997-a13fe9bd70f0: 2021-10-19T21:54:32.886Z [INFO] server-fb20e901-e794-e063-0997-a13fe9bd70f0: entering follower state: follower="Node at fb20e901-e794-e063-0997-a13fe9bd70f0 [Follower]" leader= testLoggerAdapter verbose: server-3b770e09-f291-8840-5e08-b7d89fdbed3b: 2021-10-19T21:54:32.886Z [INFO] server-3b770e09-f291-8840-5e08-b7d89fdbed3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:32.886Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-3b770e09-f291-8840-5e08-b7d89fdbed3b: 2021-10-19T21:54:32.886Z [INFO] server-3b770e09-f291-8840-5e08-b7d89fdbed3b: entering follower state: follower="Node at 3b770e09-f291-8840-5e08-b7d89fdbed3b [Follower]" leader= testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [WARN] server-bc0928e6-6a26-c775-72c3-90aee37697ef: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering candidate state: node="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Candidate]" term=2 testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: election won: tally=2 testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering leader state: leader="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Leader]" testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: added peer, starting replication: peer=server-fb20e901-e794-e063-0997-a13fe9bd70f0 testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: added peer, starting replication: peer=server-3b770e09-f291-8840-5e08-b7d89fdbed3b testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: pipelining replication: peer="{Voter server-fb20e901-e794-e063-0997-a13fe9bd70f0 fb20e901-e794-e063-0997-a13fe9bd70f0}" testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: pipelining replication: peer="{Voter server-3b770e09-f291-8840-5e08-b7d89fdbed3b 3b770e09-f291-8840-5e08-b7d89fdbed3b}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.063Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.958539182 +0000 UTC m=+14.083565394 (1 nodes), 71.815686ms from start of poll, 72.569741ms from cluster start. Timeout at 2021-10-19 21:54:33.063608682 +0000 UTC m=+14.188634807, 105.069413ms after stability testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:33.075Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: aborting pipeline replication: peer="{Voter server-3b770e09-f291-8840-5e08-b7d89fdbed3b 3b770e09-f291-8840-5e08-b7d89fdbed3b}" testLoggerAdapter verbose: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:33.075Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: aborting pipeline replication: peer="{Voter server-fb20e901-e794-e063-0997-a13fe9bd70f0 fb20e901-e794-e063-0997-a13fe9bd70f0}" --- PASS: TestRaft_Barrier (0.19s) testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.886Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.886Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering follower state: follower="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Follower]" leader= testing.go:149: server-fb20e901-e794-e063-0997-a13fe9bd70f0: 2021-10-19T21:54:32.886Z [INFO] server-fb20e901-e794-e063-0997-a13fe9bd70f0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testing.go:149: server-fb20e901-e794-e063-0997-a13fe9bd70f0: 2021-10-19T21:54:32.886Z [INFO] server-fb20e901-e794-e063-0997-a13fe9bd70f0: entering follower state: follower="Node at fb20e901-e794-e063-0997-a13fe9bd70f0 [Follower]" leader= testing.go:149: server-3b770e09-f291-8840-5e08-b7d89fdbed3b: 2021-10-19T21:54:32.886Z [INFO] server-3b770e09-f291-8840-5e08-b7d89fdbed3b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bc0928e6-6a26-c775-72c3-90aee37697ef Address:bc0928e6-6a26-c775-72c3-90aee37697ef} {Suffrage:Voter ID:server-fb20e901-e794-e063-0997-a13fe9bd70f0 Address:fb20e901-e794-e063-0997-a13fe9bd70f0} {Suffrage:Voter ID:server-3b770e09-f291-8840-5e08-b7d89fdbed3b Address:3b770e09-f291-8840-5e08-b7d89fdbed3b}]" testing.go:149: cluster: 2021-10-19T21:54:32.886Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-3b770e09-f291-8840-5e08-b7d89fdbed3b: 2021-10-19T21:54:32.886Z [INFO] server-3b770e09-f291-8840-5e08-b7d89fdbed3b: entering follower state: follower="Node at 3b770e09-f291-8840-5e08-b7d89fdbed3b [Follower]" leader= testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [WARN] server-bc0928e6-6a26-c775-72c3-90aee37697ef: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering candidate state: node="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Candidate]" term=2 testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: election won: tally=2 testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: entering leader state: leader="Node at bc0928e6-6a26-c775-72c3-90aee37697ef [Leader]" testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: added peer, starting replication: peer=server-fb20e901-e794-e063-0997-a13fe9bd70f0 testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: added peer, starting replication: peer=server-3b770e09-f291-8840-5e08-b7d89fdbed3b testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: pipelining replication: peer="{Voter server-fb20e901-e794-e063-0997-a13fe9bd70f0 fb20e901-e794-e063-0997-a13fe9bd70f0}" testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:32.958Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: pipelining replication: peer="{Voter server-3b770e09-f291-8840-5e08-b7d89fdbed3b 3b770e09-f291-8840-5e08-b7d89fdbed3b}" testing.go:149: cluster: 2021-10-19T21:54:33.063Z [INFO] stable state for Leader reached at 2021-10-19 21:54:32.958539182 +0000 UTC m=+14.083565394 (1 nodes), 71.815686ms from start of poll, 72.569741ms from cluster start. Timeout at 2021-10-19 21:54:33.063608682 +0000 UTC m=+14.188634807, 105.069413ms after stability testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:33.075Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: aborting pipeline replication: peer="{Voter server-3b770e09-f291-8840-5e08-b7d89fdbed3b 3b770e09-f291-8840-5e08-b7d89fdbed3b}" testing.go:149: server-bc0928e6-6a26-c775-72c3-90aee37697ef: 2021-10-19T21:54:33.075Z [INFO] server-bc0928e6-6a26-c775-72c3-90aee37697ef: aborting pipeline replication: peer="{Voter server-fb20e901-e794-e063-0997-a13fe9bd70f0 fb20e901-e794-e063-0997-a13fe9bd70f0}" === RUN TestRaft_VerifyLeader testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.078Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.078Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering follower state: follower="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Follower]" leader= testLoggerAdapter verbose: server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: 2021-10-19T21:54:33.078Z [INFO] server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testLoggerAdapter verbose: server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: 2021-10-19T21:54:33.078Z [INFO] server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: entering follower state: follower="Node at 602f7d68-aaf8-05e9-d6a7-14e3a64aa585 [Follower]" leader= testLoggerAdapter verbose: server-af86dda2-6856-194b-1673-c39219834362: 2021-10-19T21:54:33.078Z [INFO] server-af86dda2-6856-194b-1673-c39219834362: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.078Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-af86dda2-6856-194b-1673-c39219834362: 2021-10-19T21:54:33.078Z [INFO] server-af86dda2-6856-194b-1673-c39219834362: entering follower state: follower="Node at af86dda2-6856-194b-1673-c39219834362 [Follower]" leader= testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.141Z [WARN] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering candidate state: node="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Candidate]" term=2 testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: election won: tally=2 testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering leader state: leader="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Leader]" testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: added peer, starting replication: peer=server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: added peer, starting replication: peer=server-af86dda2-6856-194b-1673-c39219834362 testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: pipelining replication: peer="{Voter server-af86dda2-6856-194b-1673-c39219834362 af86dda2-6856-194b-1673-c39219834362}" testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: pipelining replication: peer="{Voter server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 602f7d68-aaf8-05e9-d6a7-14e3a64aa585}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.247Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.142347466 +0000 UTC m=+14.267373601 (1 nodes), 63.872646ms from start of poll, 64.91788ms from cluster start. Timeout at 2021-10-19 21:54:33.247431327 +0000 UTC m=+14.372457475, 105.083874ms after stability testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.247Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: aborting pipeline replication: peer="{Voter server-af86dda2-6856-194b-1673-c39219834362 af86dda2-6856-194b-1673-c39219834362}" testLoggerAdapter verbose: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.247Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: aborting pipeline replication: peer="{Voter server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 602f7d68-aaf8-05e9-d6a7-14e3a64aa585}" --- PASS: TestRaft_VerifyLeader (0.17s) testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.078Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.078Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering follower state: follower="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Follower]" leader= testing.go:149: server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: 2021-10-19T21:54:33.078Z [INFO] server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testing.go:149: server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: 2021-10-19T21:54:33.078Z [INFO] server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585: entering follower state: follower="Node at 602f7d68-aaf8-05e9-d6a7-14e3a64aa585 [Follower]" leader= testing.go:149: server-af86dda2-6856-194b-1673-c39219834362: 2021-10-19T21:54:33.078Z [INFO] server-af86dda2-6856-194b-1673-c39219834362: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6d35680a-7b60-3aec-10fd-fdf3998583bc Address:6d35680a-7b60-3aec-10fd-fdf3998583bc} {Suffrage:Voter ID:server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 Address:602f7d68-aaf8-05e9-d6a7-14e3a64aa585} {Suffrage:Voter ID:server-af86dda2-6856-194b-1673-c39219834362 Address:af86dda2-6856-194b-1673-c39219834362}]" testing.go:149: cluster: 2021-10-19T21:54:33.078Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-af86dda2-6856-194b-1673-c39219834362: 2021-10-19T21:54:33.078Z [INFO] server-af86dda2-6856-194b-1673-c39219834362: entering follower state: follower="Node at af86dda2-6856-194b-1673-c39219834362 [Follower]" leader= testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.141Z [WARN] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering candidate state: node="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Candidate]" term=2 testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: election won: tally=2 testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: entering leader state: leader="Node at 6d35680a-7b60-3aec-10fd-fdf3998583bc [Leader]" testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: added peer, starting replication: peer=server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: added peer, starting replication: peer=server-af86dda2-6856-194b-1673-c39219834362 testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: pipelining replication: peer="{Voter server-af86dda2-6856-194b-1673-c39219834362 af86dda2-6856-194b-1673-c39219834362}" testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.142Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: pipelining replication: peer="{Voter server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 602f7d68-aaf8-05e9-d6a7-14e3a64aa585}" testing.go:149: cluster: 2021-10-19T21:54:33.247Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.142347466 +0000 UTC m=+14.267373601 (1 nodes), 63.872646ms from start of poll, 64.91788ms from cluster start. Timeout at 2021-10-19 21:54:33.247431327 +0000 UTC m=+14.372457475, 105.083874ms after stability testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.247Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: aborting pipeline replication: peer="{Voter server-af86dda2-6856-194b-1673-c39219834362 af86dda2-6856-194b-1673-c39219834362}" testing.go:149: server-6d35680a-7b60-3aec-10fd-fdf3998583bc: 2021-10-19T21:54:33.247Z [INFO] server-6d35680a-7b60-3aec-10fd-fdf3998583bc: aborting pipeline replication: peer="{Voter server-602f7d68-aaf8-05e9-d6a7-14e3a64aa585 602f7d68-aaf8-05e9-d6a7-14e3a64aa585}" === RUN TestRaft_VerifyLeader_Single testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.249Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f Address:6ee7de8f-2b3c-9352-5ef4-1b2245f2854f}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.249Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.249Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering follower state: follower="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Follower]" leader= testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [WARN] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering candidate state: node="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Candidate]" term=2 testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: election won: tally=1 testLoggerAdapter verbose: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering leader state: leader="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.424Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.318798756 +0000 UTC m=+14.443824907 (1 nodes), 69.019477ms from start of poll, 69.285906ms from cluster start. Timeout at 2021-10-19 21:54:33.423952218 +0000 UTC m=+14.548978412, 105.153505ms after stability --- PASS: TestRaft_VerifyLeader_Single (0.18s) testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.249Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f Address:6ee7de8f-2b3c-9352-5ef4-1b2245f2854f}]" testing.go:149: cluster: 2021-10-19T21:54:33.249Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.249Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering follower state: follower="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Follower]" leader= testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [WARN] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering candidate state: node="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Candidate]" term=2 testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: election won: tally=1 testing.go:149: server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: 2021-10-19T21:54:33.318Z [INFO] server-6ee7de8f-2b3c-9352-5ef4-1b2245f2854f: entering leader state: leader="Node at 6ee7de8f-2b3c-9352-5ef4-1b2245f2854f [Leader]" testing.go:149: cluster: 2021-10-19T21:54:33.424Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.318798756 +0000 UTC m=+14.443824907 (1 nodes), 69.019477ms from start of poll, 69.285906ms from cluster start. Timeout at 2021-10-19 21:54:33.423952218 +0000 UTC m=+14.548978412, 105.153505ms after stability === RUN TestRaft_VerifyLeader_Fail testLoggerAdapter verbose: server-09092dbb-b127-040e-6443-7b136311bfdf: 2021-10-19T21:54:33.426Z [INFO] server-09092dbb-b127-040e-6443-7b136311bfdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09092dbb-b127-040e-6443-7b136311bfdf Address:09092dbb-b127-040e-6443-7b136311bfdf} {Suffrage:Voter ID:server-93f7f491-aefe-dfda-5ede-0d14f786cd65 Address:93f7f491-aefe-dfda-5ede-0d14f786cd65}]" testLoggerAdapter verbose: server-09092dbb-b127-040e-6443-7b136311bfdf: 2021-10-19T21:54:33.426Z [INFO] server-09092dbb-b127-040e-6443-7b136311bfdf: entering follower state: follower="Node at 09092dbb-b127-040e-6443-7b136311bfdf [Follower]" leader= testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.427Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09092dbb-b127-040e-6443-7b136311bfdf Address:09092dbb-b127-040e-6443-7b136311bfdf} {Suffrage:Voter ID:server-93f7f491-aefe-dfda-5ede-0d14f786cd65 Address:93f7f491-aefe-dfda-5ede-0d14f786cd65}]" testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.427Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering follower state: follower="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Follower]" leader= testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.427Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [WARN] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering candidate state: node="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Candidate]" term=2 testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: election won: tally=2 testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering leader state: leader="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Leader]" testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: added peer, starting replication: peer=server-09092dbb-b127-040e-6443-7b136311bfdf testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: pipelining replication: peer="{Voter server-09092dbb-b127-040e-6443-7b136311bfdf 09092dbb-b127-040e-6443-7b136311bfdf}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.608Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.503591156 +0000 UTC m=+14.628617296 (1 nodes), 75.701846ms from start of poll, 77.743338ms from cluster start. Timeout at 2021-10-19 21:54:33.608662218 +0000 UTC m=+14.733688366, 105.07107ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.608Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.714Z [INFO] stable state for Follower reached at 2021-10-19 21:54:33.608843476 +0000 UTC m=+14.733869618 (1 nodes), 1.063µs from start of poll, 182.99566ms from cluster start. Timeout at 2021-10-19 21:54:33.713943153 +0000 UTC m=+14.838969345, 105.099727ms after stability testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [WARN] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: new leader elected, stepping down testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering follower state: follower="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Follower]" leader= testLoggerAdapter verbose: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: aborting pipeline replication: peer="{Voter server-09092dbb-b127-040e-6443-7b136311bfdf 09092dbb-b127-040e-6443-7b136311bfdf}" --- PASS: TestRaft_VerifyLeader_Fail (0.29s) testing.go:149: server-09092dbb-b127-040e-6443-7b136311bfdf: 2021-10-19T21:54:33.426Z [INFO] server-09092dbb-b127-040e-6443-7b136311bfdf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09092dbb-b127-040e-6443-7b136311bfdf Address:09092dbb-b127-040e-6443-7b136311bfdf} {Suffrage:Voter ID:server-93f7f491-aefe-dfda-5ede-0d14f786cd65 Address:93f7f491-aefe-dfda-5ede-0d14f786cd65}]" testing.go:149: server-09092dbb-b127-040e-6443-7b136311bfdf: 2021-10-19T21:54:33.426Z [INFO] server-09092dbb-b127-040e-6443-7b136311bfdf: entering follower state: follower="Node at 09092dbb-b127-040e-6443-7b136311bfdf [Follower]" leader= testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.427Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-09092dbb-b127-040e-6443-7b136311bfdf Address:09092dbb-b127-040e-6443-7b136311bfdf} {Suffrage:Voter ID:server-93f7f491-aefe-dfda-5ede-0d14f786cd65 Address:93f7f491-aefe-dfda-5ede-0d14f786cd65}]" testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.427Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering follower state: follower="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Follower]" leader= testing.go:149: cluster: 2021-10-19T21:54:33.427Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [WARN] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering candidate state: node="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Candidate]" term=2 testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: election won: tally=2 testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering leader state: leader="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Leader]" testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: added peer, starting replication: peer=server-09092dbb-b127-040e-6443-7b136311bfdf testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.503Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: pipelining replication: peer="{Voter server-09092dbb-b127-040e-6443-7b136311bfdf 09092dbb-b127-040e-6443-7b136311bfdf}" testing.go:149: cluster: 2021-10-19T21:54:33.608Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.503591156 +0000 UTC m=+14.628617296 (1 nodes), 75.701846ms from start of poll, 77.743338ms from cluster start. Timeout at 2021-10-19 21:54:33.608662218 +0000 UTC m=+14.733688366, 105.07107ms after stability testing.go:149: cluster: 2021-10-19T21:54:33.608Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:33.714Z [INFO] stable state for Follower reached at 2021-10-19 21:54:33.608843476 +0000 UTC m=+14.733869618 (1 nodes), 1.063µs from start of poll, 182.99566ms from cluster start. Timeout at 2021-10-19 21:54:33.713943153 +0000 UTC m=+14.838969345, 105.099727ms after stability testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [WARN] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: new leader elected, stepping down testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: entering follower state: follower="Node at 93f7f491-aefe-dfda-5ede-0d14f786cd65 [Follower]" leader= testing.go:149: server-93f7f491-aefe-dfda-5ede-0d14f786cd65: 2021-10-19T21:54:33.714Z [INFO] server-93f7f491-aefe-dfda-5ede-0d14f786cd65: aborting pipeline replication: peer="{Voter server-09092dbb-b127-040e-6443-7b136311bfdf 09092dbb-b127-040e-6443-7b136311bfdf}" === RUN TestRaft_VerifyLeader_PartialConnect testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.716Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.716Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering follower state: follower="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Follower]" leader= testLoggerAdapter verbose: server-e67575a2-b09c-4278-06b4-acf2f966ccaf: 2021-10-19T21:54:33.716Z [INFO] server-e67575a2-b09c-4278-06b4-acf2f966ccaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testLoggerAdapter verbose: server-e67575a2-b09c-4278-06b4-acf2f966ccaf: 2021-10-19T21:54:33.716Z [INFO] server-e67575a2-b09c-4278-06b4-acf2f966ccaf: entering follower state: follower="Node at e67575a2-b09c-4278-06b4-acf2f966ccaf [Follower]" leader= testLoggerAdapter verbose: server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: 2021-10-19T21:54:33.716Z [INFO] server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.716Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: 2021-10-19T21:54:33.716Z [INFO] server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: entering follower state: follower="Node at caf4c5be-2001-1b0a-d6a4-750aac8c11c1 [Follower]" leader= testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [WARN] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering candidate state: node="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Candidate]" term=2 testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: election won: tally=2 testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering leader state: leader="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Leader]" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: added peer, starting replication: peer=server-e67575a2-b09c-4278-06b4-acf2f966ccaf testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: added peer, starting replication: peer=server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: pipelining replication: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: pipelining replication: peer="{Voter server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 caf4c5be-2001-1b0a-d6a4-750aac8c11c1}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.874Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.768970702 +0000 UTC m=+14.893996902 (1 nodes), 52.434364ms from start of poll, 53.122702ms from cluster start. Timeout at 2021-10-19 21:54:33.87405341 +0000 UTC m=+14.999079571, 105.082669ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.879Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:33.984Z [INFO] stable state for Follower reached at 2021-10-19 21:54:33.879453896 +0000 UTC m=+15.004480033 (2 nodes), 1.477µs from start of poll, 163.605833ms from cluster start. Timeout at 2021-10-19 21:54:33.984477449 +0000 UTC m=+15.109503573, 105.02354ms after stability testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [ERROR] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: failed to pipeline appendEntries: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" error="append pipeline closed" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: aborting pipeline replication: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [ERROR] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: failed to heartbeat to: peer=e67575a2-b09c-4278-06b4-acf2f966ccaf error="failed to connect to peer: e67575a2-b09c-4278-06b4-acf2f966ccaf" testLoggerAdapter verbose: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: aborting pipeline replication: peer="{Voter server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 caf4c5be-2001-1b0a-d6a4-750aac8c11c1}" --- PASS: TestRaft_VerifyLeader_PartialConnect (0.27s) testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.716Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.716Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering follower state: follower="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Follower]" leader= testing.go:149: server-e67575a2-b09c-4278-06b4-acf2f966ccaf: 2021-10-19T21:54:33.716Z [INFO] server-e67575a2-b09c-4278-06b4-acf2f966ccaf: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testing.go:149: server-e67575a2-b09c-4278-06b4-acf2f966ccaf: 2021-10-19T21:54:33.716Z [INFO] server-e67575a2-b09c-4278-06b4-acf2f966ccaf: entering follower state: follower="Node at e67575a2-b09c-4278-06b4-acf2f966ccaf [Follower]" leader= testing.go:149: server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: 2021-10-19T21:54:33.716Z [INFO] server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8 Address:a3cadc17-139e-5970-b21c-95fcd6a8b6c8} {Suffrage:Voter ID:server-e67575a2-b09c-4278-06b4-acf2f966ccaf Address:e67575a2-b09c-4278-06b4-acf2f966ccaf} {Suffrage:Voter ID:server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 Address:caf4c5be-2001-1b0a-d6a4-750aac8c11c1}]" testing.go:149: cluster: 2021-10-19T21:54:33.716Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: 2021-10-19T21:54:33.716Z [INFO] server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1: entering follower state: follower="Node at caf4c5be-2001-1b0a-d6a4-750aac8c11c1 [Follower]" leader= testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [WARN] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering candidate state: node="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Candidate]" term=2 testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: election won: tally=2 testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: entering leader state: leader="Node at a3cadc17-139e-5970-b21c-95fcd6a8b6c8 [Leader]" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.768Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: added peer, starting replication: peer=server-e67575a2-b09c-4278-06b4-acf2f966ccaf testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: added peer, starting replication: peer=server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: pipelining replication: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.769Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: pipelining replication: peer="{Voter server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 caf4c5be-2001-1b0a-d6a4-750aac8c11c1}" testing.go:149: cluster: 2021-10-19T21:54:33.874Z [INFO] stable state for Leader reached at 2021-10-19 21:54:33.768970702 +0000 UTC m=+14.893996902 (1 nodes), 52.434364ms from start of poll, 53.122702ms from cluster start. Timeout at 2021-10-19 21:54:33.87405341 +0000 UTC m=+14.999079571, 105.082669ms after stability testing.go:149: cluster: 2021-10-19T21:54:33.879Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:33.984Z [INFO] stable state for Follower reached at 2021-10-19 21:54:33.879453896 +0000 UTC m=+15.004480033 (2 nodes), 1.477µs from start of poll, 163.605833ms from cluster start. Timeout at 2021-10-19 21:54:33.984477449 +0000 UTC m=+15.109503573, 105.02354ms after stability raft_test.go:1647: [INFO] Disconnecting Node at e67575a2-b09c-4278-06b4-acf2f966ccaf [Follower] testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [ERROR] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: failed to pipeline appendEntries: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" error="append pipeline closed" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: aborting pipeline replication: peer="{Voter server-e67575a2-b09c-4278-06b4-acf2f966ccaf e67575a2-b09c-4278-06b4-acf2f966ccaf}" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [ERROR] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: failed to heartbeat to: peer=e67575a2-b09c-4278-06b4-acf2f966ccaf error="failed to connect to peer: e67575a2-b09c-4278-06b4-acf2f966ccaf" testing.go:149: server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: 2021-10-19T21:54:33.984Z [INFO] server-a3cadc17-139e-5970-b21c-95fcd6a8b6c8: aborting pipeline replication: peer="{Voter server-caf4c5be-2001-1b0a-d6a4-750aac8c11c1 caf4c5be-2001-1b0a-d6a4-750aac8c11c1}" === RUN TestRaft_StartAsLeader testLoggerAdapter verbose: server-2d55059f-be10-81a4-74e9-161a05660105: 2021-10-19T21:54:33.985Z [INFO] server-2d55059f-be10-81a4-74e9-161a05660105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d55059f-be10-81a4-74e9-161a05660105 Address:2d55059f-be10-81a4-74e9-161a05660105}]" testLoggerAdapter verbose: server-2d55059f-be10-81a4-74e9-161a05660105: 2021-10-19T21:54:33.986Z [INFO] server-2d55059f-be10-81a4-74e9-161a05660105: entering leader state: leader="Node at 2d55059f-be10-81a4-74e9-161a05660105 [Leader]" --- PASS: TestRaft_StartAsLeader (0.00s) testing.go:149: server-2d55059f-be10-81a4-74e9-161a05660105: 2021-10-19T21:54:33.985Z [INFO] server-2d55059f-be10-81a4-74e9-161a05660105: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-2d55059f-be10-81a4-74e9-161a05660105 Address:2d55059f-be10-81a4-74e9-161a05660105}]" testing.go:149: server-2d55059f-be10-81a4-74e9-161a05660105: 2021-10-19T21:54:33.986Z [INFO] server-2d55059f-be10-81a4-74e9-161a05660105: entering leader state: leader="Node at 2d55059f-be10-81a4-74e9-161a05660105 [Leader]" === RUN TestRaft_NotifyCh testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:33.987Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72 Address:fce45e76-6aaf-93f0-ff3e-95e54ee21d72}]" testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:33.987Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering follower state: follower="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Follower]" leader= testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [WARN] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering candidate state: node="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Candidate]" term=2 testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: election won: tally=1 testLoggerAdapter verbose: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering leader state: leader="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Leader]" --- PASS: TestRaft_NotifyCh (0.07s) testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:33.987Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72 Address:fce45e76-6aaf-93f0-ff3e-95e54ee21d72}]" testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:33.987Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering follower state: follower="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Follower]" leader= testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [WARN] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering candidate state: node="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Candidate]" term=2 testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: election won: tally=1 testing.go:149: server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: 2021-10-19T21:54:34.052Z [INFO] server-fce45e76-6aaf-93f0-ff3e-95e54ee21d72: entering leader state: leader="Node at fce45e76-6aaf-93f0-ff3e-95e54ee21d72 [Leader]" === RUN TestRaft_Voting testLoggerAdapter verbose: server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: 2021-10-19T21:54:34.054Z [INFO] server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testLoggerAdapter verbose: server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: 2021-10-19T21:54:34.054Z [INFO] server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: entering follower state: follower="Node at 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd [Follower]" leader= testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.055Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.055Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering follower state: follower="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Follower]" leader= testLoggerAdapter verbose: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.055Z [INFO] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.055Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.055Z [INFO] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: entering follower state: follower="Node at 9ceadcac-3998-fa9b-6bd6-471c6360280c [Follower]" leader= testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [WARN] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering candidate state: node="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Candidate]" term=2 testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: election won: tally=2 testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering leader state: leader="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Leader]" testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: added peer, starting replication: peer=server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: added peer, starting replication: peer=server-9ceadcac-3998-fa9b-6bd6-471c6360280c testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: pipelining replication: peer="{Voter server-9ceadcac-3998-fa9b-6bd6-471c6360280c 9ceadcac-3998-fa9b-6bd6-471c6360280c}" testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: pipelining replication: peer="{Voter server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.245Z [INFO] stable state for Follower reached at 2021-10-19 21:54:34.140421764 +0000 UTC m=+15.265447941 (2 nodes), 85.066145ms from start of poll, 85.834165ms from cluster start. Timeout at 2021-10-19 21:54:34.245517887 +0000 UTC m=+15.370544053, 105.096112ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.245Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.350Z [INFO] stable state for Leader reached at 2021-10-19 21:54:34.245742256 +0000 UTC m=+15.370768392 (1 nodes), 1.51µs from start of poll, 191.154616ms from cluster start. Timeout at 2021-10-19 21:54:34.35082969 +0000 UTC m=+15.475855827, 105.087435ms after stability testLoggerAdapter verbose: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.350Z [WARN] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: rejecting vote request since we have a leader: from=7f555bba-bb7d-8486-ad92-f1cb06d9dcdd leader=15993ecf-56c8-add9-5c1a-f1f2bf940741 testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.351Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: aborting pipeline replication: peer="{Voter server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd}" testLoggerAdapter verbose: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.351Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: aborting pipeline replication: peer="{Voter server-9ceadcac-3998-fa9b-6bd6-471c6360280c 9ceadcac-3998-fa9b-6bd6-471c6360280c}" --- PASS: TestRaft_Voting (0.30s) testing.go:149: server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: 2021-10-19T21:54:34.054Z [INFO] server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testing.go:149: server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: 2021-10-19T21:54:34.054Z [INFO] server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd: entering follower state: follower="Node at 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd [Follower]" leader= testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.055Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.055Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering follower state: follower="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Follower]" leader= testing.go:149: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.055Z [INFO] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd Address:7f555bba-bb7d-8486-ad92-f1cb06d9dcdd} {Suffrage:Voter ID:server-15993ecf-56c8-add9-5c1a-f1f2bf940741 Address:15993ecf-56c8-add9-5c1a-f1f2bf940741} {Suffrage:Voter ID:server-9ceadcac-3998-fa9b-6bd6-471c6360280c Address:9ceadcac-3998-fa9b-6bd6-471c6360280c}]" testing.go:149: cluster: 2021-10-19T21:54:34.055Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.055Z [INFO] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: entering follower state: follower="Node at 9ceadcac-3998-fa9b-6bd6-471c6360280c [Follower]" leader= testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [WARN] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering candidate state: node="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Candidate]" term=2 testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: election won: tally=2 testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: entering leader state: leader="Node at 15993ecf-56c8-add9-5c1a-f1f2bf940741 [Leader]" testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: added peer, starting replication: peer=server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: added peer, starting replication: peer=server-9ceadcac-3998-fa9b-6bd6-471c6360280c testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: pipelining replication: peer="{Voter server-9ceadcac-3998-fa9b-6bd6-471c6360280c 9ceadcac-3998-fa9b-6bd6-471c6360280c}" testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.140Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: pipelining replication: peer="{Voter server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd}" testing.go:149: cluster: 2021-10-19T21:54:34.245Z [INFO] stable state for Follower reached at 2021-10-19 21:54:34.140421764 +0000 UTC m=+15.265447941 (2 nodes), 85.066145ms from start of poll, 85.834165ms from cluster start. Timeout at 2021-10-19 21:54:34.245517887 +0000 UTC m=+15.370544053, 105.096112ms after stability testing.go:149: cluster: 2021-10-19T21:54:34.245Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:34.350Z [INFO] stable state for Leader reached at 2021-10-19 21:54:34.245742256 +0000 UTC m=+15.370768392 (1 nodes), 1.51µs from start of poll, 191.154616ms from cluster start. Timeout at 2021-10-19 21:54:34.35082969 +0000 UTC m=+15.475855827, 105.087435ms after stability testing.go:149: server-9ceadcac-3998-fa9b-6bd6-471c6360280c: 2021-10-19T21:54:34.350Z [WARN] server-9ceadcac-3998-fa9b-6bd6-471c6360280c: rejecting vote request since we have a leader: from=7f555bba-bb7d-8486-ad92-f1cb06d9dcdd leader=15993ecf-56c8-add9-5c1a-f1f2bf940741 testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.351Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: aborting pipeline replication: peer="{Voter server-7f555bba-bb7d-8486-ad92-f1cb06d9dcdd 7f555bba-bb7d-8486-ad92-f1cb06d9dcdd}" testing.go:149: server-15993ecf-56c8-add9-5c1a-f1f2bf940741: 2021-10-19T21:54:34.351Z [INFO] server-15993ecf-56c8-add9-5c1a-f1f2bf940741: aborting pipeline replication: peer="{Voter server-9ceadcac-3998-fa9b-6bd6-471c6360280c 9ceadcac-3998-fa9b-6bd6-471c6360280c}" === RUN TestRaft_ProtocolVersion_RejectRPC testLoggerAdapter verbose: server-d0312ca8-0b55-41d1-b031-997f0f279dc3: 2021-10-19T21:54:34.352Z [INFO] server-d0312ca8-0b55-41d1-b031-997f0f279dc3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testLoggerAdapter verbose: server-d0312ca8-0b55-41d1-b031-997f0f279dc3: 2021-10-19T21:54:34.353Z [INFO] server-d0312ca8-0b55-41d1-b031-997f0f279dc3: entering follower state: follower="Node at d0312ca8-0b55-41d1-b031-997f0f279dc3 [Follower]" leader= testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.353Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.353Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering follower state: follower="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Follower]" leader= testLoggerAdapter verbose: server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: 2021-10-19T21:54:34.353Z [INFO] server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.353Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: 2021-10-19T21:54:34.353Z [INFO] server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: entering follower state: follower="Node at 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad [Follower]" leader= testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [WARN] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering candidate state: node="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Candidate]" term=2 testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: election won: tally=2 testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering leader state: leader="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Leader]" testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: added peer, starting replication: peer=server-d0312ca8-0b55-41d1-b031-997f0f279dc3 testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: added peer, starting replication: peer=server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: pipelining replication: peer="{Voter server-d0312ca8-0b55-41d1-b031-997f0f279dc3 d0312ca8-0b55-41d1-b031-997f0f279dc3}" testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.416Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: pipelining replication: peer="{Voter server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.521Z [INFO] stable state for Follower reached at 2021-10-19 21:54:34.415890112 +0000 UTC m=+15.540916473 (2 nodes), 62.423871ms from start of poll, 63.087167ms from cluster start. Timeout at 2021-10-19 21:54:34.521025567 +0000 UTC m=+15.646051714, 105.135241ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.521Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:34.626Z [INFO] stable state for Leader reached at 2021-10-19 21:54:34.521313267 +0000 UTC m=+15.646339392 (1 nodes), 994ns from start of poll, 168.510086ms from cluster start. Timeout at 2021-10-19 21:54:34.62641178 +0000 UTC m=+15.751437974, 105.098582ms after stability testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.626Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: aborting pipeline replication: peer="{Voter server-d0312ca8-0b55-41d1-b031-997f0f279dc3 d0312ca8-0b55-41d1-b031-997f0f279dc3}" testLoggerAdapter verbose: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.626Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: aborting pipeline replication: peer="{Voter server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}" --- PASS: TestRaft_ProtocolVersion_RejectRPC (0.68s) testing.go:149: server-d0312ca8-0b55-41d1-b031-997f0f279dc3: 2021-10-19T21:54:34.352Z [INFO] server-d0312ca8-0b55-41d1-b031-997f0f279dc3: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testing.go:149: server-d0312ca8-0b55-41d1-b031-997f0f279dc3: 2021-10-19T21:54:34.353Z [INFO] server-d0312ca8-0b55-41d1-b031-997f0f279dc3: entering follower state: follower="Node at d0312ca8-0b55-41d1-b031-997f0f279dc3 [Follower]" leader= testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.353Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.353Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering follower state: follower="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Follower]" leader= testing.go:149: server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: 2021-10-19T21:54:34.353Z [INFO] server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d0312ca8-0b55-41d1-b031-997f0f279dc3 Address:d0312ca8-0b55-41d1-b031-997f0f279dc3} {Suffrage:Voter ID:server-acc1345f-4cce-a196-35c5-f3ced3dc472e Address:acc1345f-4cce-a196-35c5-f3ced3dc472e} {Suffrage:Voter ID:server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad Address:8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}]" testing.go:149: cluster: 2021-10-19T21:54:34.353Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: 2021-10-19T21:54:34.353Z [INFO] server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad: entering follower state: follower="Node at 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad [Follower]" leader= testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [WARN] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering candidate state: node="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Candidate]" term=2 testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: election won: tally=2 testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: entering leader state: leader="Node at acc1345f-4cce-a196-35c5-f3ced3dc472e [Leader]" testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: added peer, starting replication: peer=server-d0312ca8-0b55-41d1-b031-997f0f279dc3 testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: added peer, starting replication: peer=server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.415Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: pipelining replication: peer="{Voter server-d0312ca8-0b55-41d1-b031-997f0f279dc3 d0312ca8-0b55-41d1-b031-997f0f279dc3}" testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.416Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: pipelining replication: peer="{Voter server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}" testing.go:149: cluster: 2021-10-19T21:54:34.521Z [INFO] stable state for Follower reached at 2021-10-19 21:54:34.415890112 +0000 UTC m=+15.540916473 (2 nodes), 62.423871ms from start of poll, 63.087167ms from cluster start. Timeout at 2021-10-19 21:54:34.521025567 +0000 UTC m=+15.646051714, 105.135241ms after stability testing.go:149: cluster: 2021-10-19T21:54:34.521Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:34.626Z [INFO] stable state for Leader reached at 2021-10-19 21:54:34.521313267 +0000 UTC m=+15.646339392 (1 nodes), 994ns from start of poll, 168.510086ms from cluster start. Timeout at 2021-10-19 21:54:34.62641178 +0000 UTC m=+15.751437974, 105.098582ms after stability testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.626Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: aborting pipeline replication: peer="{Voter server-d0312ca8-0b55-41d1-b031-997f0f279dc3 d0312ca8-0b55-41d1-b031-997f0f279dc3}" testing.go:149: server-acc1345f-4cce-a196-35c5-f3ced3dc472e: 2021-10-19T21:54:34.626Z [INFO] server-acc1345f-4cce-a196-35c5-f3ced3dc472e: aborting pipeline replication: peer="{Voter server-8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad 8d3b1fb5-054d-d77b-642e-4ee9c7c0e7ad}" === RUN TestRaft_ProtocolVersion_Upgrade_1_2 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.032Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.033Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering follower state: follower="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Follower]" leader= testLoggerAdapter verbose: e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: 2021-10-19T21:54:35.033Z [INFO] e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testLoggerAdapter verbose: e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: 2021-10-19T21:54:35.033Z [INFO] e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: entering follower state: follower="Node at e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 [Follower]" leader= testLoggerAdapter verbose: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.033Z [INFO] 6a570da0-4851-1d97-51bd-3b4b1768862b: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.033Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.033Z [INFO] 6a570da0-4851-1d97-51bd-3b4b1768862b: entering follower state: follower="Node at 6a570da0-4851-1d97-51bd-3b4b1768862b [Follower]" leader= testLoggerAdapter verbose: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.117Z [WARN] 6a570da0-4851-1d97-51bd-3b4b1768862b: no known peers, aborting election testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.132Z [WARN] 5b47f744-29a0-635f-f9f2-c933bd69fa63: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.132Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering candidate state: node="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Candidate]" term=2 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: election won: tally=2 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering leader state: leader="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Leader]" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: added peer, starting replication: peer=e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: pipelining replication: peer="{Voter e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.238Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.133660341 +0000 UTC m=+16.258686478 (1 nodes), 99.845582ms from start of poll, 100.894836ms from cluster start. Timeout at 2021-10-19 21:54:35.238769489 +0000 UTC m=+16.363795631, 105.109153ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.238Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.344Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.239043319 +0000 UTC m=+16.364069447 (1 nodes), 1.619µs from start of poll, 206.277805ms from cluster start. Timeout at 2021-10-19 21:54:35.344195534 +0000 UTC m=+16.469221685, 105.152238ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.344Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.449Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.344452222 +0000 UTC m=+16.469478356 (1 nodes), 1.266µs from start of poll, 311.686714ms from cluster start. Timeout at 2021-10-19 21:54:35.449598377 +0000 UTC m=+16.574624518, 105.146162ms after stability testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.449Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: updating configuration: command=AddStaging server-id=6a570da0-4851-1d97-51bd-3b4b1768862b server-addr=6a570da0-4851-1d97-51bd-3b4b1768862b servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5} {Suffrage:Voter ID:6a570da0-4851-1d97-51bd-3b4b1768862b Address:6a570da0-4851-1d97-51bd-3b4b1768862b}]" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: added peer, starting replication: peer=6a570da0-4851-1d97-51bd-3b4b1768862b testLoggerAdapter verbose: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.450Z [WARN] 6a570da0-4851-1d97-51bd-3b4b1768862b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [WARN] 5b47f744-29a0-635f-f9f2-c933bd69fa63: appendEntries rejected, sending older logs: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" next=1 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: pipelining replication: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.455Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.560Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.455472451 +0000 UTC m=+16.580498573 (1 nodes), 793ns from start of poll, 422.706931ms from cluster start. Timeout at 2021-10-19 21:54:35.560540557 +0000 UTC m=+16.685566707, 105.068134ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.560Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.665Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.560730497 +0000 UTC m=+16.685756633 (1 nodes), 1.035µs from start of poll, 527.964991ms from cluster start. Timeout at 2021-10-19 21:54:35.66581293 +0000 UTC m=+16.790839086, 105.082453ms after stability testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: updating configuration: command=RemoveServer server-id=6a570da0-4851-1d97-51bd-3b4b1768862b server-addr= servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: removed peer, stopping replication: peer=6a570da0-4851-1d97-51bd-3b4b1768862b last-index=4 testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: aborting pipeline replication: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" testLoggerAdapter verbose: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: aborting pipeline replication: peer="{Voter e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}" --- PASS: TestRaft_ProtocolVersion_Upgrade_1_2 (0.64s) testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.032Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.033Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering follower state: follower="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Follower]" leader= testing.go:149: e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: 2021-10-19T21:54:35.033Z [INFO] e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: initial configuration: index=1 servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testing.go:149: e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: 2021-10-19T21:54:35.033Z [INFO] e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5: entering follower state: follower="Node at e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 [Follower]" leader= testing.go:149: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.033Z [INFO] 6a570da0-4851-1d97-51bd-3b4b1768862b: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-19T21:54:35.033Z [INFO] starting stability test: raft-state=Leader testing.go:149: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.033Z [INFO] 6a570da0-4851-1d97-51bd-3b4b1768862b: entering follower state: follower="Node at 6a570da0-4851-1d97-51bd-3b4b1768862b [Follower]" leader= testing.go:149: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.117Z [WARN] 6a570da0-4851-1d97-51bd-3b4b1768862b: no known peers, aborting election testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.132Z [WARN] 5b47f744-29a0-635f-f9f2-c933bd69fa63: heartbeat timeout reached, starting election: last-leader= testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.132Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering candidate state: node="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Candidate]" term=2 testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: election won: tally=2 testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: entering leader state: leader="Node at 5b47f744-29a0-635f-f9f2-c933bd69fa63 [Leader]" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: added peer, starting replication: peer=e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.133Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: pipelining replication: peer="{Voter e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}" testing.go:149: cluster: 2021-10-19T21:54:35.238Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.133660341 +0000 UTC m=+16.258686478 (1 nodes), 99.845582ms from start of poll, 100.894836ms from cluster start. Timeout at 2021-10-19 21:54:35.238769489 +0000 UTC m=+16.363795631, 105.109153ms after stability testing.go:149: cluster: 2021-10-19T21:54:35.238Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:35.344Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.239043319 +0000 UTC m=+16.364069447 (1 nodes), 1.619µs from start of poll, 206.277805ms from cluster start. Timeout at 2021-10-19 21:54:35.344195534 +0000 UTC m=+16.469221685, 105.152238ms after stability testing.go:149: cluster: 2021-10-19T21:54:35.344Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:35.449Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.344452222 +0000 UTC m=+16.469478356 (1 nodes), 1.266µs from start of poll, 311.686714ms from cluster start. Timeout at 2021-10-19 21:54:35.449598377 +0000 UTC m=+16.574624518, 105.146162ms after stability testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.449Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: updating configuration: command=AddStaging server-id=6a570da0-4851-1d97-51bd-3b4b1768862b server-addr=6a570da0-4851-1d97-51bd-3b4b1768862b servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5} {Suffrage:Voter ID:6a570da0-4851-1d97-51bd-3b4b1768862b Address:6a570da0-4851-1d97-51bd-3b4b1768862b}]" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: added peer, starting replication: peer=6a570da0-4851-1d97-51bd-3b4b1768862b testing.go:149: 6a570da0-4851-1d97-51bd-3b4b1768862b: 2021-10-19T21:54:35.450Z [WARN] 6a570da0-4851-1d97-51bd-3b4b1768862b: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [WARN] 5b47f744-29a0-635f-f9f2-c933bd69fa63: appendEntries rejected, sending older logs: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" next=1 testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.450Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: pipelining replication: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" testing.go:149: cluster: 2021-10-19T21:54:35.455Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:35.560Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.455472451 +0000 UTC m=+16.580498573 (1 nodes), 793ns from start of poll, 422.706931ms from cluster start. Timeout at 2021-10-19 21:54:35.560540557 +0000 UTC m=+16.685566707, 105.068134ms after stability testing.go:149: cluster: 2021-10-19T21:54:35.560Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:35.665Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.560730497 +0000 UTC m=+16.685756633 (1 nodes), 1.035µs from start of poll, 527.964991ms from cluster start. Timeout at 2021-10-19 21:54:35.66581293 +0000 UTC m=+16.790839086, 105.082453ms after stability testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: updating configuration: command=RemoveServer server-id=6a570da0-4851-1d97-51bd-3b4b1768862b server-addr= servers="[{Suffrage:Voter ID:5b47f744-29a0-635f-f9f2-c933bd69fa63 Address:5b47f744-29a0-635f-f9f2-c933bd69fa63} {Suffrage:Voter ID:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 Address:e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}]" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: removed peer, stopping replication: peer=6a570da0-4851-1d97-51bd-3b4b1768862b last-index=4 testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: aborting pipeline replication: peer="{Voter 6a570da0-4851-1d97-51bd-3b4b1768862b 6a570da0-4851-1d97-51bd-3b4b1768862b}" testing.go:149: 5b47f744-29a0-635f-f9f2-c933bd69fa63: 2021-10-19T21:54:35.666Z [INFO] 5b47f744-29a0-635f-f9f2-c933bd69fa63: aborting pipeline replication: peer="{Voter e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5 e02c26f5-17fd-9fd6-d4e8-ecac4d2d2ab5}" === RUN TestRaft_ProtocolVersion_Upgrade_2_3 testLoggerAdapter verbose: 200bb8e6-7d0b-d492-1300-d7ff74a940f6: 2021-10-19T21:54:35.668Z [INFO] 200bb8e6-7d0b-d492-1300-d7ff74a940f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4}]" testLoggerAdapter verbose: 200bb8e6-7d0b-d492-1300-d7ff74a940f6: 2021-10-19T21:54:35.668Z [INFO] 200bb8e6-7d0b-d492-1300-d7ff74a940f6: entering follower state: follower="Node at 200bb8e6-7d0b-d492-1300-d7ff74a940f6 [Follower]" leader= testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.668Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.668Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.668Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering follower state: follower="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Follower]" leader= testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [WARN] e8e65781-3267-19b8-a77b-68a6da5f4ab4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering candidate state: node="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Candidate]" term=2 testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: election won: tally=2 testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering leader state: leader="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Leader]" testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: added peer, starting replication: peer=200bb8e6-7d0b-d492-1300-d7ff74a940f6 testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: pipelining replication: peer="{Voter 200bb8e6-7d0b-d492-1300-d7ff74a940f6 200bb8e6-7d0b-d492-1300-d7ff74a940f6}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.825Z [INFO] stable state for Follower reached at 2021-10-19 21:54:35.720674722 +0000 UTC m=+16.845700914 (1 nodes), 51.85234ms from start of poll, 52.38114ms from cluster start. Timeout at 2021-10-19 21:54:35.825787931 +0000 UTC m=+16.950814119, 105.113205ms after stability testLoggerAdapter verbose: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.827Z [INFO] server-5db9ea98-eb33-e5f1-9279-ca383843f171: initial configuration: index=0 servers=[] testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.827Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.827Z [INFO] server-5db9ea98-eb33-e5f1-9279-ca383843f171: entering follower state: follower="Node at 5db9ea98-eb33-e5f1-9279-ca383843f171 [Follower]" leader= testLoggerAdapter verbose: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.890Z [WARN] server-5db9ea98-eb33-e5f1-9279-ca383843f171: no known peers, aborting election testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.932Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.827176918 +0000 UTC m=+16.952203040 (1 nodes), 770ns from start of poll, 158.883266ms from cluster start. Timeout at 2021-10-19 21:54:35.932297177 +0000 UTC m=+17.057323318, 105.120278ms after stability testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.932Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: updating configuration: command=AddStaging server-id=server-5db9ea98-eb33-e5f1-9279-ca383843f171 server-addr=5db9ea98-eb33-e5f1-9279-ca383843f171 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4} {Suffrage:Voter ID:server-5db9ea98-eb33-e5f1-9279-ca383843f171 Address:5db9ea98-eb33-e5f1-9279-ca383843f171}]" testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.932Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: added peer, starting replication: peer=server-5db9ea98-eb33-e5f1-9279-ca383843f171 testLoggerAdapter verbose: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.933Z [WARN] server-5db9ea98-eb33-e5f1-9279-ca383843f171: failed to get previous log: previous-index=3 last-index=0 error="log not found" testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.933Z [WARN] e8e65781-3267-19b8-a77b-68a6da5f4ab4: appendEntries rejected, sending older logs: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" next=1 testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.933Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: pipelining replication: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:35.938Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.043Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.938275312 +0000 UTC m=+17.063301443 (1 nodes), 1.34µs from start of poll, 269.981669ms from cluster start. Timeout at 2021-10-19 21:54:36.043357365 +0000 UTC m=+17.168383525, 105.082082ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.043Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.148Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.043629887 +0000 UTC m=+17.168656015 (1 nodes), 1.117µs from start of poll, 375.336241ms from cluster start. Timeout at 2021-10-19 21:54:36.148755127 +0000 UTC m=+17.273781280, 105.125265ms after stability testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.148Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: updating configuration: command=RemoveServer server-id=200bb8e6-7d0b-d492-1300-d7ff74a940f6 server-addr= servers="[{Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4} {Suffrage:Voter ID:server-5db9ea98-eb33-e5f1-9279-ca383843f171 Address:5db9ea98-eb33-e5f1-9279-ca383843f171}]" testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: removed peer, stopping replication: peer=200bb8e6-7d0b-d492-1300-d7ff74a940f6 last-index=4 testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: aborting pipeline replication: peer="{Voter 200bb8e6-7d0b-d492-1300-d7ff74a940f6 200bb8e6-7d0b-d492-1300-d7ff74a940f6}" testLoggerAdapter verbose: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: aborting pipeline replication: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" --- PASS: TestRaft_ProtocolVersion_Upgrade_2_3 (0.48s) testing.go:149: 200bb8e6-7d0b-d492-1300-d7ff74a940f6: 2021-10-19T21:54:35.668Z [INFO] 200bb8e6-7d0b-d492-1300-d7ff74a940f6: initial configuration: index=1 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4}]" testing.go:149: 200bb8e6-7d0b-d492-1300-d7ff74a940f6: 2021-10-19T21:54:35.668Z [INFO] 200bb8e6-7d0b-d492-1300-d7ff74a940f6: entering follower state: follower="Node at 200bb8e6-7d0b-d492-1300-d7ff74a940f6 [Follower]" leader= testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.668Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: initial configuration: index=1 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4}]" testing.go:149: cluster: 2021-10-19T21:54:35.668Z [INFO] starting stability test: raft-state=Follower testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.668Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering follower state: follower="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Follower]" leader= testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [WARN] e8e65781-3267-19b8-a77b-68a6da5f4ab4: heartbeat timeout reached, starting election: last-leader= testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering candidate state: node="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Candidate]" term=2 testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: election won: tally=2 testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: entering leader state: leader="Node at e8e65781-3267-19b8-a77b-68a6da5f4ab4 [Leader]" testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: added peer, starting replication: peer=200bb8e6-7d0b-d492-1300-d7ff74a940f6 testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.720Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: pipelining replication: peer="{Voter 200bb8e6-7d0b-d492-1300-d7ff74a940f6 200bb8e6-7d0b-d492-1300-d7ff74a940f6}" testing.go:149: cluster: 2021-10-19T21:54:35.825Z [INFO] stable state for Follower reached at 2021-10-19 21:54:35.720674722 +0000 UTC m=+16.845700914 (1 nodes), 51.85234ms from start of poll, 52.38114ms from cluster start. Timeout at 2021-10-19 21:54:35.825787931 +0000 UTC m=+16.950814119, 105.113205ms after stability testing.go:149: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.827Z [INFO] server-5db9ea98-eb33-e5f1-9279-ca383843f171: initial configuration: index=0 servers=[] testing.go:149: cluster: 2021-10-19T21:54:35.827Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.827Z [INFO] server-5db9ea98-eb33-e5f1-9279-ca383843f171: entering follower state: follower="Node at 5db9ea98-eb33-e5f1-9279-ca383843f171 [Follower]" leader= testing.go:149: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.890Z [WARN] server-5db9ea98-eb33-e5f1-9279-ca383843f171: no known peers, aborting election testing.go:149: cluster: 2021-10-19T21:54:35.932Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.827176918 +0000 UTC m=+16.952203040 (1 nodes), 770ns from start of poll, 158.883266ms from cluster start. Timeout at 2021-10-19 21:54:35.932297177 +0000 UTC m=+17.057323318, 105.120278ms after stability testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.932Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: updating configuration: command=AddStaging server-id=server-5db9ea98-eb33-e5f1-9279-ca383843f171 server-addr=5db9ea98-eb33-e5f1-9279-ca383843f171 servers="[{Suffrage:Voter ID:200bb8e6-7d0b-d492-1300-d7ff74a940f6 Address:200bb8e6-7d0b-d492-1300-d7ff74a940f6} {Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4} {Suffrage:Voter ID:server-5db9ea98-eb33-e5f1-9279-ca383843f171 Address:5db9ea98-eb33-e5f1-9279-ca383843f171}]" testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.932Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: added peer, starting replication: peer=server-5db9ea98-eb33-e5f1-9279-ca383843f171 testing.go:149: server-5db9ea98-eb33-e5f1-9279-ca383843f171: 2021-10-19T21:54:35.933Z [WARN] server-5db9ea98-eb33-e5f1-9279-ca383843f171: failed to get previous log: previous-index=3 last-index=0 error="log not found" testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.933Z [WARN] e8e65781-3267-19b8-a77b-68a6da5f4ab4: appendEntries rejected, sending older logs: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" next=1 testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:35.933Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: pipelining replication: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" testing.go:149: cluster: 2021-10-19T21:54:35.938Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:36.043Z [INFO] stable state for Leader reached at 2021-10-19 21:54:35.938275312 +0000 UTC m=+17.063301443 (1 nodes), 1.34µs from start of poll, 269.981669ms from cluster start. Timeout at 2021-10-19 21:54:36.043357365 +0000 UTC m=+17.168383525, 105.082082ms after stability testing.go:149: cluster: 2021-10-19T21:54:36.043Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:36.148Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.043629887 +0000 UTC m=+17.168656015 (1 nodes), 1.117µs from start of poll, 375.336241ms from cluster start. Timeout at 2021-10-19 21:54:36.148755127 +0000 UTC m=+17.273781280, 105.125265ms after stability testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.148Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: updating configuration: command=RemoveServer server-id=200bb8e6-7d0b-d492-1300-d7ff74a940f6 server-addr= servers="[{Suffrage:Voter ID:e8e65781-3267-19b8-a77b-68a6da5f4ab4 Address:e8e65781-3267-19b8-a77b-68a6da5f4ab4} {Suffrage:Voter ID:server-5db9ea98-eb33-e5f1-9279-ca383843f171 Address:5db9ea98-eb33-e5f1-9279-ca383843f171}]" testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: removed peer, stopping replication: peer=200bb8e6-7d0b-d492-1300-d7ff74a940f6 last-index=4 testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: aborting pipeline replication: peer="{Voter 200bb8e6-7d0b-d492-1300-d7ff74a940f6 200bb8e6-7d0b-d492-1300-d7ff74a940f6}" testing.go:149: e8e65781-3267-19b8-a77b-68a6da5f4ab4: 2021-10-19T21:54:36.149Z [INFO] e8e65781-3267-19b8-a77b-68a6da5f4ab4: aborting pipeline replication: peer="{Voter server-5db9ea98-eb33-e5f1-9279-ca383843f171 5db9ea98-eb33-e5f1-9279-ca383843f171}" === RUN TestRaft_LeadershipTransferInProgress --- PASS: TestRaft_LeadershipTransferInProgress (0.00s) === RUN TestRaft_LeadershipTransferPickServer --- PASS: TestRaft_LeadershipTransferPickServer (0.00s) === RUN TestRaft_LeadershipTransfer testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.151Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.151Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering follower state: follower="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Follower]" leader= testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.151Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.152Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering follower state: follower="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Follower]" leader= testLoggerAdapter verbose: server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: 2021-10-19T21:54:36.152Z [INFO] server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.152Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: 2021-10-19T21:54:36.152Z [INFO] server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: entering follower state: follower="Node at f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 [Follower]" leader= testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.202Z [WARN] server-bdf19c9c-7091-ca25-595c-d615f5852558: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering candidate state: node="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Candidate]" term=2 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: election won: tally=2 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering leader state: leader="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Leader]" testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: added peer, starting replication: peer=server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: added peer, starting replication: peer=server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.204Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: pipelining replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.204Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: pipelining replication: peer="{Voter server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 9d3ab266-99c0-7939-4547-f39cb6ae4b65}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.309Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.203947624 +0000 UTC m=+17.328973794 (1 nodes), 51.659645ms from start of poll, 52.425893ms from cluster start. Timeout at 2021-10-19 21:54:36.309030022 +0000 UTC m=+17.434056182, 105.082388ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.309Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.414Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.30926264 +0000 UTC m=+17.434288764 (1 nodes), 1.081µs from start of poll, 157.740863ms from cluster start. Timeout at 2021-10-19 21:54:36.414375784 +0000 UTC m=+17.539401978, 105.113214ms after stability testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.414Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering candidate state: node="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.414Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.414Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: election won: tally=2 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering follower state: follower="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Follower]" leader= testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering leader state: leader="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Leader]" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: added peer, starting replication: peer=server-bdf19c9c-7091-ca25-595c-d615f5852558 testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: added peer, starting replication: peer=server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: aborting pipeline replication: peer="{Voter server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 9d3ab266-99c0-7939-4547-f39cb6ae4b65}" testLoggerAdapter verbose: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: aborting pipeline replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: pipelining replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: pipelining replication: peer="{Voter server-bdf19c9c-7091-ca25-595c-d615f5852558 bdf19c9c-7091-ca25-595c-d615f5852558}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.520Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.415048929 +0000 UTC m=+17.540075062 (1 nodes), 104.381µs from start of poll, 263.527161ms from cluster start. Timeout at 2021-10-19 21:54:36.520164402 +0000 UTC m=+17.645190580, 105.115518ms after stability testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.520Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: aborting pipeline replication: peer="{Voter server-bdf19c9c-7091-ca25-595c-d615f5852558 bdf19c9c-7091-ca25-595c-d615f5852558}" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.520Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: aborting pipeline replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testLoggerAdapter verbose: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.570Z [ERROR] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: failed to heartbeat to: peer=f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 error="command timed out" --- PASS: TestRaft_LeadershipTransfer (0.42s) testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.151Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.151Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering follower state: follower="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Follower]" leader= testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.151Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.152Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering follower state: follower="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Follower]" leader= testing.go:149: server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: 2021-10-19T21:54:36.152Z [INFO] server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 Address:9d3ab266-99c0-7939-4547-f39cb6ae4b65} {Suffrage:Voter ID:server-bdf19c9c-7091-ca25-595c-d615f5852558 Address:bdf19c9c-7091-ca25-595c-d615f5852558} {Suffrage:Voter ID:server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 Address:f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}]" testing.go:149: cluster: 2021-10-19T21:54:36.152Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: 2021-10-19T21:54:36.152Z [INFO] server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5: entering follower state: follower="Node at f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 [Follower]" leader= testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.202Z [WARN] server-bdf19c9c-7091-ca25-595c-d615f5852558: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering candidate state: node="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Candidate]" term=2 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: election won: tally=2 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering leader state: leader="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Leader]" testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: added peer, starting replication: peer=server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.203Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: added peer, starting replication: peer=server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.204Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: pipelining replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.204Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: pipelining replication: peer="{Voter server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 9d3ab266-99c0-7939-4547-f39cb6ae4b65}" testing.go:149: cluster: 2021-10-19T21:54:36.309Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.203947624 +0000 UTC m=+17.328973794 (1 nodes), 51.659645ms from start of poll, 52.425893ms from cluster start. Timeout at 2021-10-19 21:54:36.309030022 +0000 UTC m=+17.434056182, 105.082388ms after stability testing.go:149: cluster: 2021-10-19T21:54:36.309Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:36.414Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.30926264 +0000 UTC m=+17.434288764 (1 nodes), 1.081µs from start of poll, 157.740863ms from cluster start. Timeout at 2021-10-19 21:54:36.414375784 +0000 UTC m=+17.539401978, 105.113214ms after stability testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.414Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering candidate state: node="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Candidate]" term=3 testing.go:149: cluster: 2021-10-19T21:54:36.414Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.414Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: election won: tally=2 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: entering follower state: follower="Node at bdf19c9c-7091-ca25-595c-d615f5852558 [Follower]" leader= testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: entering leader state: leader="Node at 9d3ab266-99c0-7939-4547-f39cb6ae4b65 [Leader]" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: added peer, starting replication: peer=server-bdf19c9c-7091-ca25-595c-d615f5852558 testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: added peer, starting replication: peer=server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: aborting pipeline replication: peer="{Voter server-9d3ab266-99c0-7939-4547-f39cb6ae4b65 9d3ab266-99c0-7939-4547-f39cb6ae4b65}" testing.go:149: server-bdf19c9c-7091-ca25-595c-d615f5852558: 2021-10-19T21:54:36.414Z [INFO] server-bdf19c9c-7091-ca25-595c-d615f5852558: aborting pipeline replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: pipelining replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.415Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: pipelining replication: peer="{Voter server-bdf19c9c-7091-ca25-595c-d615f5852558 bdf19c9c-7091-ca25-595c-d615f5852558}" testing.go:149: cluster: 2021-10-19T21:54:36.520Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.415048929 +0000 UTC m=+17.540075062 (1 nodes), 104.381µs from start of poll, 263.527161ms from cluster start. Timeout at 2021-10-19 21:54:36.520164402 +0000 UTC m=+17.645190580, 105.115518ms after stability testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.520Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: aborting pipeline replication: peer="{Voter server-bdf19c9c-7091-ca25-595c-d615f5852558 bdf19c9c-7091-ca25-595c-d615f5852558}" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.520Z [INFO] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: aborting pipeline replication: peer="{Voter server-f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 f7c1b825-4525-8cc4-6cf8-00d870ffe5b5}" testing.go:149: server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: 2021-10-19T21:54:36.570Z [ERROR] server-9d3ab266-99c0-7939-4547-f39cb6ae4b65: failed to heartbeat to: peer=f7c1b825-4525-8cc4-6cf8-00d870ffe5b5 error="command timed out" === RUN TestRaft_LeadershipTransferWithOneNode testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.572Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2cfb4d4-a479-b773-660c-ad2323da30da Address:b2cfb4d4-a479-b773-660c-ad2323da30da}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.573Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.573Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering follower state: follower="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Follower]" leader= testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [WARN] server-b2cfb4d4-a479-b773-660c-ad2323da30da: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering candidate state: node="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Candidate]" term=2 testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: election won: tally=1 testLoggerAdapter verbose: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering leader state: leader="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Leader]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.778Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.673052182 +0000 UTC m=+17.798078356 (1 nodes), 99.962777ms from start of poll, 100.380098ms from cluster start. Timeout at 2021-10-19 21:54:36.778138234 +0000 UTC m=+17.903164387, 105.086031ms after stability --- PASS: TestRaft_LeadershipTransferWithOneNode (0.21s) testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.572Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b2cfb4d4-a479-b773-660c-ad2323da30da Address:b2cfb4d4-a479-b773-660c-ad2323da30da}]" testing.go:149: cluster: 2021-10-19T21:54:36.573Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.573Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering follower state: follower="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Follower]" leader= testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [WARN] server-b2cfb4d4-a479-b773-660c-ad2323da30da: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering candidate state: node="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Candidate]" term=2 testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: election won: tally=1 testing.go:149: server-b2cfb4d4-a479-b773-660c-ad2323da30da: 2021-10-19T21:54:36.672Z [INFO] server-b2cfb4d4-a479-b773-660c-ad2323da30da: entering leader state: leader="Node at b2cfb4d4-a479-b773-660c-ad2323da30da [Leader]" testing.go:149: cluster: 2021-10-19T21:54:36.778Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.673052182 +0000 UTC m=+17.798078356 (1 nodes), 99.962777ms from start of poll, 100.380098ms from cluster start. Timeout at 2021-10-19 21:54:36.778138234 +0000 UTC m=+17.903164387, 105.086031ms after stability === RUN TestRaft_LeadershipTransferWithSevenNodes testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:36.780Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:36.780Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering follower state: follower="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Follower]" leader= testLoggerAdapter verbose: server-956d65b6-e982-d08a-7433-9a3ec893cae9: 2021-10-19T21:54:36.780Z [INFO] server-956d65b6-e982-d08a-7433-9a3ec893cae9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-956d65b6-e982-d08a-7433-9a3ec893cae9: 2021-10-19T21:54:36.780Z [INFO] server-956d65b6-e982-d08a-7433-9a3ec893cae9: entering follower state: follower="Node at 956d65b6-e982-d08a-7433-9a3ec893cae9 [Follower]" leader= testLoggerAdapter verbose: server-0fef5354-0b52-5969-30d3-641076237458: 2021-10-19T21:54:36.781Z [INFO] server-0fef5354-0b52-5969-30d3-641076237458: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-0fef5354-0b52-5969-30d3-641076237458: 2021-10-19T21:54:36.781Z [INFO] server-0fef5354-0b52-5969-30d3-641076237458: entering follower state: follower="Node at 0fef5354-0b52-5969-30d3-641076237458 [Follower]" leader= testLoggerAdapter verbose: server-377a4fb6-be23-2658-a52f-84d4d49a2a00: 2021-10-19T21:54:36.781Z [INFO] server-377a4fb6-be23-2658-a52f-84d4d49a2a00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-377a4fb6-be23-2658-a52f-84d4d49a2a00: 2021-10-19T21:54:36.781Z [INFO] server-377a4fb6-be23-2658-a52f-84d4d49a2a00: entering follower state: follower="Node at 377a4fb6-be23-2658-a52f-84d4d49a2a00 [Follower]" leader= testLoggerAdapter verbose: server-85724c3b-4491-0750-2d11-5de7f27c3e54: 2021-10-19T21:54:36.781Z [INFO] server-85724c3b-4491-0750-2d11-5de7f27c3e54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-85724c3b-4491-0750-2d11-5de7f27c3e54: 2021-10-19T21:54:36.781Z [INFO] server-85724c3b-4491-0750-2d11-5de7f27c3e54: entering follower state: follower="Node at 85724c3b-4491-0750-2d11-5de7f27c3e54 [Follower]" leader= testLoggerAdapter verbose: server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: 2021-10-19T21:54:36.781Z [INFO] server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: 2021-10-19T21:54:36.782Z [INFO] server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: entering follower state: follower="Node at 9ffa28c2-c8d3-0d76-7435-890f415e6ad5 [Follower]" leader= testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.782Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.782Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.782Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering follower state: follower="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Follower]" leader= testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.832Z [WARN] server-658de050-f839-1788-bd3f-3240f79184b4: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.832Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering candidate state: node="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Candidate]" term=2 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: election won: tally=4 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering leader state: leader="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Leader]" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-fad81140-af39-3a0c-3733-b9f1723ddd06 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-956d65b6-e982-d08a-7433-9a3ec893cae9 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-0fef5354-0b52-5969-30d3-641076237458 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-377a4fb6-be23-2658-a52f-84d4d49a2a00 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-85724c3b-4491-0750-2d11-5de7f27c3e54 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-fad81140-af39-3a0c-3733-b9f1723ddd06 fad81140-af39-3a0c-3733-b9f1723ddd06}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.938Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.833271479 +0000 UTC m=+17.958297610 (1 nodes), 51.049452ms from start of poll, 52.845457ms from cluster start. Timeout at 2021-10-19 21:54:36.938352265 +0000 UTC m=+18.063378437, 105.080827ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:36.938Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.043Z [INFO] stable state for Follower reached at 2021-10-19 21:54:36.938560059 +0000 UTC m=+18.063586215 (6 nodes), 2.946µs from start of poll, 158.134062ms from cluster start. Timeout at 2021-10-19 21:54:37.043641865 +0000 UTC m=+18.168668048, 105.081833ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.044Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.149Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.044693082 +0000 UTC m=+18.169719253 (1 nodes), 2.529µs from start of poll, 264.2671ms from cluster start. Timeout at 2021-10-19 21:54:37.1497032 +0000 UTC m=+18.274729324, 105.010071ms after stability testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.149Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering candidate state: node="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Candidate]" term=3 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering follower state: follower="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Follower]" leader= testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.150Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: election won: tally=4 testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering leader state: leader="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Leader]" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-956d65b6-e982-d08a-7433-9a3ec893cae9 testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-0fef5354-0b52-5969-30d3-641076237458 testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-377a4fb6-be23-2658-a52f-84d4d49a2a00 testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-85724c3b-4491-0750-2d11-5de7f27c3e54 testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-658de050-f839-1788-bd3f-3240f79184b4 testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-fad81140-af39-3a0c-3733-b9f1723ddd06 fad81140-af39-3a0c-3733-b9f1723ddd06}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testLoggerAdapter verbose: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-658de050-f839-1788-bd3f-3240f79184b4 658de050-f839-1788-bd3f-3240f79184b4}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.255Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.150760812 +0000 UTC m=+18.275786963 (1 nodes), 428.084µs from start of poll, 370.33481ms from cluster start. Timeout at 2021-10-19 21:54:37.255850902 +0000 UTC m=+18.380877076, 105.090113ms after stability testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-658de050-f839-1788-bd3f-3240f79184b4 658de050-f839-1788-bd3f-3240f79184b4}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testLoggerAdapter verbose: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" --- PASS: TestRaft_LeadershipTransferWithSevenNodes (0.48s) testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:36.780Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:36.780Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering follower state: follower="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Follower]" leader= testing.go:149: server-956d65b6-e982-d08a-7433-9a3ec893cae9: 2021-10-19T21:54:36.780Z [INFO] server-956d65b6-e982-d08a-7433-9a3ec893cae9: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-956d65b6-e982-d08a-7433-9a3ec893cae9: 2021-10-19T21:54:36.780Z [INFO] server-956d65b6-e982-d08a-7433-9a3ec893cae9: entering follower state: follower="Node at 956d65b6-e982-d08a-7433-9a3ec893cae9 [Follower]" leader= testing.go:149: server-0fef5354-0b52-5969-30d3-641076237458: 2021-10-19T21:54:36.781Z [INFO] server-0fef5354-0b52-5969-30d3-641076237458: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-0fef5354-0b52-5969-30d3-641076237458: 2021-10-19T21:54:36.781Z [INFO] server-0fef5354-0b52-5969-30d3-641076237458: entering follower state: follower="Node at 0fef5354-0b52-5969-30d3-641076237458 [Follower]" leader= testing.go:149: server-377a4fb6-be23-2658-a52f-84d4d49a2a00: 2021-10-19T21:54:36.781Z [INFO] server-377a4fb6-be23-2658-a52f-84d4d49a2a00: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-377a4fb6-be23-2658-a52f-84d4d49a2a00: 2021-10-19T21:54:36.781Z [INFO] server-377a4fb6-be23-2658-a52f-84d4d49a2a00: entering follower state: follower="Node at 377a4fb6-be23-2658-a52f-84d4d49a2a00 [Follower]" leader= testing.go:149: server-85724c3b-4491-0750-2d11-5de7f27c3e54: 2021-10-19T21:54:36.781Z [INFO] server-85724c3b-4491-0750-2d11-5de7f27c3e54: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-85724c3b-4491-0750-2d11-5de7f27c3e54: 2021-10-19T21:54:36.781Z [INFO] server-85724c3b-4491-0750-2d11-5de7f27c3e54: entering follower state: follower="Node at 85724c3b-4491-0750-2d11-5de7f27c3e54 [Follower]" leader= testing.go:149: server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: 2021-10-19T21:54:36.781Z [INFO] server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: 2021-10-19T21:54:36.782Z [INFO] server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5: entering follower state: follower="Node at 9ffa28c2-c8d3-0d76-7435-890f415e6ad5 [Follower]" leader= testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.782Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-fad81140-af39-3a0c-3733-b9f1723ddd06 Address:fad81140-af39-3a0c-3733-b9f1723ddd06} {Suffrage:Voter ID:server-956d65b6-e982-d08a-7433-9a3ec893cae9 Address:956d65b6-e982-d08a-7433-9a3ec893cae9} {Suffrage:Voter ID:server-0fef5354-0b52-5969-30d3-641076237458 Address:0fef5354-0b52-5969-30d3-641076237458} {Suffrage:Voter ID:server-377a4fb6-be23-2658-a52f-84d4d49a2a00 Address:377a4fb6-be23-2658-a52f-84d4d49a2a00} {Suffrage:Voter ID:server-85724c3b-4491-0750-2d11-5de7f27c3e54 Address:85724c3b-4491-0750-2d11-5de7f27c3e54} {Suffrage:Voter ID:server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 Address:9ffa28c2-c8d3-0d76-7435-890f415e6ad5} {Suffrage:Voter ID:server-658de050-f839-1788-bd3f-3240f79184b4 Address:658de050-f839-1788-bd3f-3240f79184b4}]" testing.go:149: cluster: 2021-10-19T21:54:36.782Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.782Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering follower state: follower="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Follower]" leader= testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.832Z [WARN] server-658de050-f839-1788-bd3f-3240f79184b4: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.832Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering candidate state: node="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Candidate]" term=2 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: election won: tally=4 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering leader state: leader="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Leader]" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-fad81140-af39-3a0c-3733-b9f1723ddd06 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-956d65b6-e982-d08a-7433-9a3ec893cae9 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-0fef5354-0b52-5969-30d3-641076237458 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-377a4fb6-be23-2658-a52f-84d4d49a2a00 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-85724c3b-4491-0750-2d11-5de7f27c3e54 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: added peer, starting replication: peer=server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-fad81140-af39-3a0c-3733-b9f1723ddd06 fad81140-af39-3a0c-3733-b9f1723ddd06}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:36.833Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: pipelining replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testing.go:149: cluster: 2021-10-19T21:54:36.938Z [INFO] stable state for Leader reached at 2021-10-19 21:54:36.833271479 +0000 UTC m=+17.958297610 (1 nodes), 51.049452ms from start of poll, 52.845457ms from cluster start. Timeout at 2021-10-19 21:54:36.938352265 +0000 UTC m=+18.063378437, 105.080827ms after stability testing.go:149: cluster: 2021-10-19T21:54:36.938Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:37.043Z [INFO] stable state for Follower reached at 2021-10-19 21:54:36.938560059 +0000 UTC m=+18.063586215 (6 nodes), 2.946µs from start of poll, 158.134062ms from cluster start. Timeout at 2021-10-19 21:54:37.043641865 +0000 UTC m=+18.168668048, 105.081833ms after stability testing.go:149: cluster: 2021-10-19T21:54:37.044Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:37.149Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.044693082 +0000 UTC m=+18.169719253 (1 nodes), 2.529µs from start of poll, 264.2671ms from cluster start. Timeout at 2021-10-19 21:54:37.1497032 +0000 UTC m=+18.274729324, 105.010071ms after stability testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.149Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering candidate state: node="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Candidate]" term=3 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: entering follower state: follower="Node at 658de050-f839-1788-bd3f-3240f79184b4 [Follower]" leader= testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testing.go:149: cluster: 2021-10-19T21:54:37.150Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: election won: tally=4 testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: entering leader state: leader="Node at fad81140-af39-3a0c-3733-b9f1723ddd06 [Leader]" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-956d65b6-e982-d08a-7433-9a3ec893cae9 testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-0fef5354-0b52-5969-30d3-641076237458 testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-377a4fb6-be23-2658-a52f-84d4d49a2a00 testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-85724c3b-4491-0750-2d11-5de7f27c3e54 testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: added peer, starting replication: peer=server-658de050-f839-1788-bd3f-3240f79184b4 testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-fad81140-af39-3a0c-3733-b9f1723ddd06 fad81140-af39-3a0c-3733-b9f1723ddd06}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testing.go:149: server-658de050-f839-1788-bd3f-3240f79184b4: 2021-10-19T21:54:37.150Z [INFO] server-658de050-f839-1788-bd3f-3240f79184b4: aborting pipeline replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.150Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: pipelining replication: peer="{Voter server-658de050-f839-1788-bd3f-3240f79184b4 658de050-f839-1788-bd3f-3240f79184b4}" testing.go:149: cluster: 2021-10-19T21:54:37.255Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.150760812 +0000 UTC m=+18.275786963 (1 nodes), 428.084µs from start of poll, 370.33481ms from cluster start. Timeout at 2021-10-19 21:54:37.255850902 +0000 UTC m=+18.380877076, 105.090113ms after stability testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-85724c3b-4491-0750-2d11-5de7f27c3e54 85724c3b-4491-0750-2d11-5de7f27c3e54}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-658de050-f839-1788-bd3f-3240f79184b4 658de050-f839-1788-bd3f-3240f79184b4}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-0fef5354-0b52-5969-30d3-641076237458 0fef5354-0b52-5969-30d3-641076237458}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-956d65b6-e982-d08a-7433-9a3ec893cae9 956d65b6-e982-d08a-7433-9a3ec893cae9}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-9ffa28c2-c8d3-0d76-7435-890f415e6ad5 9ffa28c2-c8d3-0d76-7435-890f415e6ad5}" testing.go:149: server-fad81140-af39-3a0c-3733-b9f1723ddd06: 2021-10-19T21:54:37.256Z [INFO] server-fad81140-af39-3a0c-3733-b9f1723ddd06: aborting pipeline replication: peer="{Voter server-377a4fb6-be23-2658-a52f-84d4d49a2a00 377a4fb6-be23-2658-a52f-84d4d49a2a00}" === RUN TestRaft_LeadershipTransferToInvalidID testLoggerAdapter verbose: server-446ae6aa-c9ed-23d0-7c3f-02917177df65: 2021-10-19T21:54:37.259Z [INFO] server-446ae6aa-c9ed-23d0-7c3f-02917177df65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testLoggerAdapter verbose: server-446ae6aa-c9ed-23d0-7c3f-02917177df65: 2021-10-19T21:54:37.259Z [INFO] server-446ae6aa-c9ed-23d0-7c3f-02917177df65: entering follower state: follower="Node at 446ae6aa-c9ed-23d0-7c3f-02917177df65 [Follower]" leader= testLoggerAdapter verbose: server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: 2021-10-19T21:54:37.259Z [INFO] server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testLoggerAdapter verbose: server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: 2021-10-19T21:54:37.259Z [INFO] server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: entering follower state: follower="Node at db5267f5-ef00-ec27-c337-2c83bf0c4a30 [Follower]" leader= testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.259Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.259Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.259Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering follower state: follower="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Follower]" leader= testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [WARN] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering candidate state: node="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Candidate]" term=2 testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: election won: tally=2 testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering leader state: leader="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Leader]" testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: added peer, starting replication: peer=server-446ae6aa-c9ed-23d0-7c3f-02917177df65 testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: added peer, starting replication: peer=server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: pipelining replication: peer="{Voter server-446ae6aa-c9ed-23d0-7c3f-02917177df65 446ae6aa-c9ed-23d0-7c3f-02917177df65}" testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: pipelining replication: peer="{Voter server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 db5267f5-ef00-ec27-c337-2c83bf0c4a30}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.424Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.319104702 +0000 UTC m=+18.444130827 (1 nodes), 59.2784ms from start of poll, 59.967624ms from cluster start. Timeout at 2021-10-19 21:54:37.424196901 +0000 UTC m=+18.549223095, 105.092268ms after stability testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.424Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: aborting pipeline replication: peer="{Voter server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 db5267f5-ef00-ec27-c337-2c83bf0c4a30}" testLoggerAdapter verbose: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.424Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: aborting pipeline replication: peer="{Voter server-446ae6aa-c9ed-23d0-7c3f-02917177df65 446ae6aa-c9ed-23d0-7c3f-02917177df65}" --- PASS: TestRaft_LeadershipTransferToInvalidID (0.17s) testing.go:149: server-446ae6aa-c9ed-23d0-7c3f-02917177df65: 2021-10-19T21:54:37.259Z [INFO] server-446ae6aa-c9ed-23d0-7c3f-02917177df65: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testing.go:149: server-446ae6aa-c9ed-23d0-7c3f-02917177df65: 2021-10-19T21:54:37.259Z [INFO] server-446ae6aa-c9ed-23d0-7c3f-02917177df65: entering follower state: follower="Node at 446ae6aa-c9ed-23d0-7c3f-02917177df65 [Follower]" leader= testing.go:149: server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: 2021-10-19T21:54:37.259Z [INFO] server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testing.go:149: server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: 2021-10-19T21:54:37.259Z [INFO] server-db5267f5-ef00-ec27-c337-2c83bf0c4a30: entering follower state: follower="Node at db5267f5-ef00-ec27-c337-2c83bf0c4a30 [Follower]" leader= testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.259Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-446ae6aa-c9ed-23d0-7c3f-02917177df65 Address:446ae6aa-c9ed-23d0-7c3f-02917177df65} {Suffrage:Voter ID:server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 Address:db5267f5-ef00-ec27-c337-2c83bf0c4a30} {Suffrage:Voter ID:server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 Address:70e35dc1-0a2f-4cfe-e930-663e3ab58ff8}]" testing.go:149: cluster: 2021-10-19T21:54:37.259Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.259Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering follower state: follower="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Follower]" leader= testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [WARN] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering candidate state: node="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Candidate]" term=2 testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.318Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: election won: tally=2 testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: entering leader state: leader="Node at 70e35dc1-0a2f-4cfe-e930-663e3ab58ff8 [Leader]" testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: added peer, starting replication: peer=server-446ae6aa-c9ed-23d0-7c3f-02917177df65 testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: added peer, starting replication: peer=server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: pipelining replication: peer="{Voter server-446ae6aa-c9ed-23d0-7c3f-02917177df65 446ae6aa-c9ed-23d0-7c3f-02917177df65}" testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.319Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: pipelining replication: peer="{Voter server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 db5267f5-ef00-ec27-c337-2c83bf0c4a30}" testing.go:149: cluster: 2021-10-19T21:54:37.424Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.319104702 +0000 UTC m=+18.444130827 (1 nodes), 59.2784ms from start of poll, 59.967624ms from cluster start. Timeout at 2021-10-19 21:54:37.424196901 +0000 UTC m=+18.549223095, 105.092268ms after stability testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.424Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: aborting pipeline replication: peer="{Voter server-db5267f5-ef00-ec27-c337-2c83bf0c4a30 db5267f5-ef00-ec27-c337-2c83bf0c4a30}" testing.go:149: server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: 2021-10-19T21:54:37.424Z [INFO] server-70e35dc1-0a2f-4cfe-e930-663e3ab58ff8: aborting pipeline replication: peer="{Voter server-446ae6aa-c9ed-23d0-7c3f-02917177df65 446ae6aa-c9ed-23d0-7c3f-02917177df65}" === RUN TestRaft_LeadershipTransferToInvalidAddress testLoggerAdapter verbose: server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: 2021-10-19T21:54:37.427Z [INFO] server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.427Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testLoggerAdapter verbose: server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: 2021-10-19T21:54:37.427Z [INFO] server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: entering follower state: follower="Node at bede58dd-ddf6-3c67-535d-be3c5eb846fe [Follower]" leader= testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.427Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering follower state: follower="Node at 84369cb4-d84e-a364-5598-99876c917691 [Follower]" leader= testLoggerAdapter verbose: server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: 2021-10-19T21:54:37.427Z [INFO] server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.427Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: 2021-10-19T21:54:37.427Z [INFO] server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: entering follower state: follower="Node at 7d39a3c2-84fa-0861-ee26-df20f6a49ad0 [Follower]" leader= testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [WARN] server-84369cb4-d84e-a364-5598-99876c917691: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering candidate state: node="Node at 84369cb4-d84e-a364-5598-99876c917691 [Candidate]" term=2 testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: election won: tally=2 testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering leader state: leader="Node at 84369cb4-d84e-a364-5598-99876c917691 [Leader]" testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: added peer, starting replication: peer=server-bede58dd-ddf6-3c67-535d-be3c5eb846fe testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: added peer, starting replication: peer=server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: pipelining replication: peer="{Voter server-bede58dd-ddf6-3c67-535d-be3c5eb846fe bede58dd-ddf6-3c67-535d-be3c5eb846fe}" testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: pipelining replication: peer="{Voter server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 7d39a3c2-84fa-0861-ee26-df20f6a49ad0}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.602Z [INFO] stable state for Follower reached at 2021-10-19 21:54:37.497275434 +0000 UTC m=+18.622301561 (2 nodes), 69.641323ms from start of poll, 70.250148ms from cluster start. Timeout at 2021-10-19 21:54:37.602359571 +0000 UTC m=+18.727385733, 105.084172ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.602Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.707Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.602545869 +0000 UTC m=+18.727571994 (1 nodes), 896ns from start of poll, 175.520581ms from cluster start. Timeout at 2021-10-19 21:54:37.707674509 +0000 UTC m=+18.832700672, 105.128678ms after stability testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.707Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: aborting pipeline replication: peer="{Voter server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 7d39a3c2-84fa-0861-ee26-df20f6a49ad0}" testLoggerAdapter verbose: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.708Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: aborting pipeline replication: peer="{Voter server-bede58dd-ddf6-3c67-535d-be3c5eb846fe bede58dd-ddf6-3c67-535d-be3c5eb846fe}" --- PASS: TestRaft_LeadershipTransferToInvalidAddress (0.28s) testing.go:149: server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: 2021-10-19T21:54:37.427Z [INFO] server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.427Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testing.go:149: server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: 2021-10-19T21:54:37.427Z [INFO] server-bede58dd-ddf6-3c67-535d-be3c5eb846fe: entering follower state: follower="Node at bede58dd-ddf6-3c67-535d-be3c5eb846fe [Follower]" leader= testing.go:149: server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: 2021-10-19T21:54:37.427Z [INFO] server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-bede58dd-ddf6-3c67-535d-be3c5eb846fe Address:bede58dd-ddf6-3c67-535d-be3c5eb846fe} {Suffrage:Voter ID:server-84369cb4-d84e-a364-5598-99876c917691 Address:84369cb4-d84e-a364-5598-99876c917691} {Suffrage:Voter ID:server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 Address:7d39a3c2-84fa-0861-ee26-df20f6a49ad0}]" testing.go:149: cluster: 2021-10-19T21:54:37.427Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.427Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering follower state: follower="Node at 84369cb4-d84e-a364-5598-99876c917691 [Follower]" leader= testing.go:149: server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: 2021-10-19T21:54:37.427Z [INFO] server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0: entering follower state: follower="Node at 7d39a3c2-84fa-0861-ee26-df20f6a49ad0 [Follower]" leader= testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [WARN] server-84369cb4-d84e-a364-5598-99876c917691: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering candidate state: node="Node at 84369cb4-d84e-a364-5598-99876c917691 [Candidate]" term=2 testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: election won: tally=2 testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: entering leader state: leader="Node at 84369cb4-d84e-a364-5598-99876c917691 [Leader]" testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: added peer, starting replication: peer=server-bede58dd-ddf6-3c67-535d-be3c5eb846fe testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: added peer, starting replication: peer=server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: pipelining replication: peer="{Voter server-bede58dd-ddf6-3c67-535d-be3c5eb846fe bede58dd-ddf6-3c67-535d-be3c5eb846fe}" testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.497Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: pipelining replication: peer="{Voter server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 7d39a3c2-84fa-0861-ee26-df20f6a49ad0}" testing.go:149: cluster: 2021-10-19T21:54:37.602Z [INFO] stable state for Follower reached at 2021-10-19 21:54:37.497275434 +0000 UTC m=+18.622301561 (2 nodes), 69.641323ms from start of poll, 70.250148ms from cluster start. Timeout at 2021-10-19 21:54:37.602359571 +0000 UTC m=+18.727385733, 105.084172ms after stability testing.go:149: cluster: 2021-10-19T21:54:37.602Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:37.707Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.602545869 +0000 UTC m=+18.727571994 (1 nodes), 896ns from start of poll, 175.520581ms from cluster start. Timeout at 2021-10-19 21:54:37.707674509 +0000 UTC m=+18.832700672, 105.128678ms after stability testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.707Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: aborting pipeline replication: peer="{Voter server-7d39a3c2-84fa-0861-ee26-df20f6a49ad0 7d39a3c2-84fa-0861-ee26-df20f6a49ad0}" testing.go:149: server-84369cb4-d84e-a364-5598-99876c917691: 2021-10-19T21:54:37.708Z [INFO] server-84369cb4-d84e-a364-5598-99876c917691: aborting pipeline replication: peer="{Voter server-bede58dd-ddf6-3c67-535d-be3c5eb846fe bede58dd-ddf6-3c67-535d-be3c5eb846fe}" === RUN TestRaft_LeadershipTransferToBehindServer testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.710Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.710Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering follower state: follower="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Follower]" leader= testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:37.710Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:37.710Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering follower state: follower="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Follower]" leader= testLoggerAdapter verbose: server-d9e5d770-1056-807a-862f-2e1b068dabf1: 2021-10-19T21:54:37.710Z [INFO] server-d9e5d770-1056-807a-862f-2e1b068dabf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.710Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-d9e5d770-1056-807a-862f-2e1b068dabf1: 2021-10-19T21:54:37.710Z [INFO] server-d9e5d770-1056-807a-862f-2e1b068dabf1: entering follower state: follower="Node at d9e5d770-1056-807a-862f-2e1b068dabf1 [Follower]" leader= testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.772Z [WARN] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering candidate state: node="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Candidate]" term=2 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: election won: tally=2 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering leader state: leader="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Leader]" testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: added peer, starting replication: peer=server-b83ff729-d29d-3792-ca40-d010d984c693 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: added peer, starting replication: peer=server-d9e5d770-1056-807a-862f-2e1b068dabf1 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: pipelining replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: pipelining replication: peer="{Voter server-b83ff729-d29d-3792-ca40-d010d984c693 b83ff729-d29d-3792-ca40-d010d984c693}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.878Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.773297598 +0000 UTC m=+18.898323765 (1 nodes), 62.333232ms from start of poll, 63.529841ms from cluster start. Timeout at 2021-10-19 21:54:37.878391458 +0000 UTC m=+19.003417611, 105.093846ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.878Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:37.983Z [INFO] stable state for Follower reached at 2021-10-19 21:54:37.878566274 +0000 UTC m=+19.003592397 (2 nodes), 967ns from start of poll, 168.798473ms from cluster start. Timeout at 2021-10-19 21:54:37.983636029 +0000 UTC m=+19.108662170, 105.069773ms after stability testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.047Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering candidate state: node="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Candidate]" term=3 testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.047Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.047Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: election won: tally=2 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering follower state: follower="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Follower]" leader= testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering leader state: leader="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Leader]" testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: aborting pipeline replication: peer="{Voter server-b83ff729-d29d-3792-ca40-d010d984c693 b83ff729-d29d-3792-ca40-d010d984c693}" testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: added peer, starting replication: peer=server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: added peer, starting replication: peer=server-d9e5d770-1056-807a-862f-2e1b068dabf1 testLoggerAdapter verbose: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: aborting pipeline replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: pipelining replication: peer="{Voter server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 d3efaca6-ffae-6226-aa28-f240a79fa8c5}" testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: pipelining replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.153Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.048087454 +0000 UTC m=+19.173113596 (1 nodes), 199.733µs from start of poll, 338.319672ms from cluster start. Timeout at 2021-10-19 21:54:38.153158694 +0000 UTC m=+19.278184832, 105.071236ms after stability testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.153Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: aborting pipeline replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testLoggerAdapter verbose: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.153Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: aborting pipeline replication: peer="{Voter server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 d3efaca6-ffae-6226-aa28-f240a79fa8c5}" --- PASS: TestRaft_LeadershipTransferToBehindServer (0.45s) testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.710Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.710Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering follower state: follower="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Follower]" leader= testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:37.710Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:37.710Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering follower state: follower="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Follower]" leader= testing.go:149: server-d9e5d770-1056-807a-862f-2e1b068dabf1: 2021-10-19T21:54:37.710Z [INFO] server-d9e5d770-1056-807a-862f-2e1b068dabf1: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 Address:d3efaca6-ffae-6226-aa28-f240a79fa8c5} {Suffrage:Voter ID:server-b83ff729-d29d-3792-ca40-d010d984c693 Address:b83ff729-d29d-3792-ca40-d010d984c693} {Suffrage:Voter ID:server-d9e5d770-1056-807a-862f-2e1b068dabf1 Address:d9e5d770-1056-807a-862f-2e1b068dabf1}]" testing.go:149: cluster: 2021-10-19T21:54:37.710Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-d9e5d770-1056-807a-862f-2e1b068dabf1: 2021-10-19T21:54:37.710Z [INFO] server-d9e5d770-1056-807a-862f-2e1b068dabf1: entering follower state: follower="Node at d9e5d770-1056-807a-862f-2e1b068dabf1 [Follower]" leader= testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.772Z [WARN] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering candidate state: node="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Candidate]" term=2 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: election won: tally=2 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering leader state: leader="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Leader]" testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: added peer, starting replication: peer=server-b83ff729-d29d-3792-ca40-d010d984c693 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: added peer, starting replication: peer=server-d9e5d770-1056-807a-862f-2e1b068dabf1 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: pipelining replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:37.773Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: pipelining replication: peer="{Voter server-b83ff729-d29d-3792-ca40-d010d984c693 b83ff729-d29d-3792-ca40-d010d984c693}" testing.go:149: cluster: 2021-10-19T21:54:37.878Z [INFO] stable state for Leader reached at 2021-10-19 21:54:37.773297598 +0000 UTC m=+18.898323765 (1 nodes), 62.333232ms from start of poll, 63.529841ms from cluster start. Timeout at 2021-10-19 21:54:37.878391458 +0000 UTC m=+19.003417611, 105.093846ms after stability testing.go:149: cluster: 2021-10-19T21:54:37.878Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:37.983Z [INFO] stable state for Follower reached at 2021-10-19 21:54:37.878566274 +0000 UTC m=+19.003592397 (2 nodes), 967ns from start of poll, 168.798473ms from cluster start. Timeout at 2021-10-19 21:54:37.983636029 +0000 UTC m=+19.108662170, 105.069773ms after stability testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.047Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering candidate state: node="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Candidate]" term=3 testing.go:149: cluster: 2021-10-19T21:54:38.047Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.047Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: election won: tally=2 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: entering follower state: follower="Node at d3efaca6-ffae-6226-aa28-f240a79fa8c5 [Follower]" leader= testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: entering leader state: leader="Node at b83ff729-d29d-3792-ca40-d010d984c693 [Leader]" testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: aborting pipeline replication: peer="{Voter server-b83ff729-d29d-3792-ca40-d010d984c693 b83ff729-d29d-3792-ca40-d010d984c693}" testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: added peer, starting replication: peer=server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: added peer, starting replication: peer=server-d9e5d770-1056-807a-862f-2e1b068dabf1 testing.go:149: server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: 2021-10-19T21:54:38.047Z [INFO] server-d3efaca6-ffae-6226-aa28-f240a79fa8c5: aborting pipeline replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: pipelining replication: peer="{Voter server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 d3efaca6-ffae-6226-aa28-f240a79fa8c5}" testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.048Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: pipelining replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testing.go:149: cluster: 2021-10-19T21:54:38.153Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.048087454 +0000 UTC m=+19.173113596 (1 nodes), 199.733µs from start of poll, 338.319672ms from cluster start. Timeout at 2021-10-19 21:54:38.153158694 +0000 UTC m=+19.278184832, 105.071236ms after stability testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.153Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: aborting pipeline replication: peer="{Voter server-d9e5d770-1056-807a-862f-2e1b068dabf1 d9e5d770-1056-807a-862f-2e1b068dabf1}" testing.go:149: server-b83ff729-d29d-3792-ca40-d010d984c693: 2021-10-19T21:54:38.153Z [INFO] server-b83ff729-d29d-3792-ca40-d010d984c693: aborting pipeline replication: peer="{Voter server-d3efaca6-ffae-6226-aa28-f240a79fa8c5 d3efaca6-ffae-6226-aa28-f240a79fa8c5}" === RUN TestRaft_LeadershipTransferToItself testLoggerAdapter verbose: server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: 2021-10-19T21:54:38.156Z [INFO] server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testLoggerAdapter verbose: server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: 2021-10-19T21:54:38.156Z [INFO] server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: entering follower state: follower="Node at 5563ae94-bb99-1ceb-736a-40774f8d5fc4 [Follower]" leader= testLoggerAdapter verbose: server-c58d3157-eb36-224b-8154-552a1d99aa1b: 2021-10-19T21:54:38.156Z [INFO] server-c58d3157-eb36-224b-8154-552a1d99aa1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testLoggerAdapter verbose: server-c58d3157-eb36-224b-8154-552a1d99aa1b: 2021-10-19T21:54:38.156Z [INFO] server-c58d3157-eb36-224b-8154-552a1d99aa1b: entering follower state: follower="Node at c58d3157-eb36-224b-8154-552a1d99aa1b [Follower]" leader= testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.156Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.156Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.157Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering follower state: follower="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Follower]" leader= testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [WARN] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering candidate state: node="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Candidate]" term=2 testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: election won: tally=2 testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering leader state: leader="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Leader]" testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: added peer, starting replication: peer=server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: added peer, starting replication: peer=server-c58d3157-eb36-224b-8154-552a1d99aa1b testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: pipelining replication: peer="{Voter server-c58d3157-eb36-224b-8154-552a1d99aa1b c58d3157-eb36-224b-8154-552a1d99aa1b}" testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: pipelining replication: peer="{Voter server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 5563ae94-bb99-1ceb-736a-40774f8d5fc4}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.317Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.212502459 +0000 UTC m=+19.337528593 (1 nodes), 55.461508ms from start of poll, 56.685876ms from cluster start. Timeout at 2021-10-19 21:54:38.317588763 +0000 UTC m=+19.442614954, 105.086361ms after stability testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: cannot transfer leadership to itself testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: aborting pipeline replication: peer="{Voter server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 5563ae94-bb99-1ceb-736a-40774f8d5fc4}" testLoggerAdapter verbose: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: aborting pipeline replication: peer="{Voter server-c58d3157-eb36-224b-8154-552a1d99aa1b c58d3157-eb36-224b-8154-552a1d99aa1b}" --- PASS: TestRaft_LeadershipTransferToItself (0.16s) testing.go:149: server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: 2021-10-19T21:54:38.156Z [INFO] server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testing.go:149: server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: 2021-10-19T21:54:38.156Z [INFO] server-5563ae94-bb99-1ceb-736a-40774f8d5fc4: entering follower state: follower="Node at 5563ae94-bb99-1ceb-736a-40774f8d5fc4 [Follower]" leader= testing.go:149: server-c58d3157-eb36-224b-8154-552a1d99aa1b: 2021-10-19T21:54:38.156Z [INFO] server-c58d3157-eb36-224b-8154-552a1d99aa1b: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testing.go:149: server-c58d3157-eb36-224b-8154-552a1d99aa1b: 2021-10-19T21:54:38.156Z [INFO] server-c58d3157-eb36-224b-8154-552a1d99aa1b: entering follower state: follower="Node at c58d3157-eb36-224b-8154-552a1d99aa1b [Follower]" leader= testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.156Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 Address:5563ae94-bb99-1ceb-736a-40774f8d5fc4} {Suffrage:Voter ID:server-c58d3157-eb36-224b-8154-552a1d99aa1b Address:c58d3157-eb36-224b-8154-552a1d99aa1b} {Suffrage:Voter ID:server-a9ab7cfd-cadf-c844-f103-6aefd670fc58 Address:a9ab7cfd-cadf-c844-f103-6aefd670fc58}]" testing.go:149: cluster: 2021-10-19T21:54:38.156Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.157Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering follower state: follower="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Follower]" leader= testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [WARN] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering candidate state: node="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Candidate]" term=2 testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: election won: tally=2 testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: entering leader state: leader="Node at a9ab7cfd-cadf-c844-f103-6aefd670fc58 [Leader]" testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: added peer, starting replication: peer=server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: added peer, starting replication: peer=server-c58d3157-eb36-224b-8154-552a1d99aa1b testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: pipelining replication: peer="{Voter server-c58d3157-eb36-224b-8154-552a1d99aa1b c58d3157-eb36-224b-8154-552a1d99aa1b}" testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.212Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: pipelining replication: peer="{Voter server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 5563ae94-bb99-1ceb-736a-40774f8d5fc4}" testing.go:149: cluster: 2021-10-19T21:54:38.317Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.212502459 +0000 UTC m=+19.337528593 (1 nodes), 55.461508ms from start of poll, 56.685876ms from cluster start. Timeout at 2021-10-19 21:54:38.317588763 +0000 UTC m=+19.442614954, 105.086361ms after stability testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: cannot transfer leadership to itself testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: aborting pipeline replication: peer="{Voter server-5563ae94-bb99-1ceb-736a-40774f8d5fc4 5563ae94-bb99-1ceb-736a-40774f8d5fc4}" testing.go:149: server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: 2021-10-19T21:54:38.317Z [INFO] server-a9ab7cfd-cadf-c844-f103-6aefd670fc58: aborting pipeline replication: peer="{Voter server-c58d3157-eb36-224b-8154-552a1d99aa1b c58d3157-eb36-224b-8154-552a1d99aa1b}" === RUN TestRaft_LeadershipTransferLeaderRejectsClientRequests --- SKIP: TestRaft_LeadershipTransferLeaderRejectsClientRequests (0.00s) raft_test.go:2064: Disabled unreliable test === RUN TestRaft_LeadershipTransferLeaderReplicationTimeout testLoggerAdapter verbose: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.321Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testLoggerAdapter verbose: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.321Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: entering follower state: follower="Node at b4aa59c4-c124-5230-4930-a189d330efa0 [Follower]" leader= testLoggerAdapter verbose: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.321Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testLoggerAdapter verbose: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.321Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: entering follower state: follower="Node at 5ef43523-94a4-651e-1b30-bbc4ced84e32 [Follower]" leader= testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.321Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.322Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.322Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering follower state: follower="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Follower]" leader= testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering candidate state: node="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Candidate]" term=2 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: election won: tally=2 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering leader state: leader="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Leader]" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: added peer, starting replication: peer=server-b4aa59c4-c124-5230-4930-a189d330efa0 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: added peer, starting replication: peer=server-5ef43523-94a4-651e-1b30-bbc4ced84e32 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-b4aa59c4-c124-5230-4930-a189d330efa0 b4aa59c4-c124-5230-4930-a189d330efa0}" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.483Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.378798802 +0000 UTC m=+19.503824938 (1 nodes), 56.671342ms from start of poll, 57.767097ms from cluster start. Timeout at 2021-10-19 21:54:38.483871353 +0000 UTC m=+19.608897504, 105.072566ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.484Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.589Z [INFO] stable state for Follower reached at 2021-10-19 21:54:38.48408946 +0000 UTC m=+19.609115598 (2 nodes), 1.673µs from start of poll, 163.057757ms from cluster start. Timeout at 2021-10-19 21:54:38.589169944 +0000 UTC m=+19.714196087, 105.080489ms after stability testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.736Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to contact: server-id=server-b4aa59c4-c124-5230-4930-a189d330efa0 time=78.959142ms testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.806Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to heartbeat to: peer=5ef43523-94a4-651e-1b30-bbc4ced84e32 error="send timed out" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.831Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to pipeline appendEntries: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" error="command enqueue timeout" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.833Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.836Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testLoggerAdapter verbose: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.929Z [WARN] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: heartbeat timeout reached, starting election: last-leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testLoggerAdapter verbose: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.930Z [WARN] server-b4aa59c4-c124-5230-4930-a189d330efa0: heartbeat timeout reached, starting election: last-leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.930Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to contact: server-id=server-b4aa59c4-c124-5230-4930-a189d330efa0 time=68.459823ms testLoggerAdapter verbose: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.933Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: entering candidate state: node="Node at 5ef43523-94a4-651e-1b30-bbc4ced84e32 [Candidate]" term=3 testLoggerAdapter verbose: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.933Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: entering candidate state: node="Node at b4aa59c4-c124-5230-4930-a189d330efa0 [Candidate]" term=3 testLoggerAdapter verbose: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.934Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: peer has newer term, stopping replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testLoggerAdapter verbose: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.934Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: duplicate requestVote for same term: term=3 testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: rejecting vote request since we have a leader: from=5ef43523-94a4-651e-1b30-bbc4ced84e32 leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.935Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: rejecting vote request since we have a leader: from=b4aa59c4-c124-5230-4930-a189d330efa0 leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.930Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-b4aa59c4-c124-5230-4930-a189d330efa0 b4aa59c4-c124-5230-4930-a189d330efa0}" testLoggerAdapter verbose: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.935Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering follower state: follower="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Follower]" leader= --- FAIL: TestRaft_LeadershipTransferLeaderReplicationTimeout (0.64s) testing.go:149: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.321Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testing.go:149: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.321Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: entering follower state: follower="Node at b4aa59c4-c124-5230-4930-a189d330efa0 [Follower]" leader= testing.go:149: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.321Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testing.go:149: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.321Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: entering follower state: follower="Node at 5ef43523-94a4-651e-1b30-bbc4ced84e32 [Follower]" leader= testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.321Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-b4aa59c4-c124-5230-4930-a189d330efa0 Address:b4aa59c4-c124-5230-4930-a189d330efa0} {Suffrage:Voter ID:server-5ef43523-94a4-651e-1b30-bbc4ced84e32 Address:5ef43523-94a4-651e-1b30-bbc4ced84e32} {Suffrage:Voter ID:server-11f81a5b-c3cb-a8f3-c73a-953965727a6f Address:11f81a5b-c3cb-a8f3-c73a-953965727a6f}]" testing.go:149: cluster: 2021-10-19T21:54:38.322Z [INFO] starting stability test: raft-state=Leader testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.322Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering follower state: follower="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Follower]" leader= testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering candidate state: node="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Candidate]" term=2 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: election won: tally=2 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering leader state: leader="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Leader]" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: added peer, starting replication: peer=server-b4aa59c4-c124-5230-4930-a189d330efa0 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: added peer, starting replication: peer=server-5ef43523-94a4-651e-1b30-bbc4ced84e32 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-b4aa59c4-c124-5230-4930-a189d330efa0 b4aa59c4-c124-5230-4930-a189d330efa0}" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.378Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testing.go:149: cluster: 2021-10-19T21:54:38.483Z [INFO] stable state for Leader reached at 2021-10-19 21:54:38.378798802 +0000 UTC m=+19.503824938 (1 nodes), 56.671342ms from start of poll, 57.767097ms from cluster start. Timeout at 2021-10-19 21:54:38.483871353 +0000 UTC m=+19.608897504, 105.072566ms after stability testing.go:149: cluster: 2021-10-19T21:54:38.484Z [INFO] starting stability test: raft-state=Follower testing.go:149: cluster: 2021-10-19T21:54:38.589Z [INFO] stable state for Follower reached at 2021-10-19 21:54:38.48408946 +0000 UTC m=+19.609115598 (2 nodes), 1.673µs from start of poll, 163.057757ms from cluster start. Timeout at 2021-10-19 21:54:38.589169944 +0000 UTC m=+19.714196087, 105.080489ms after stability testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.736Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to contact: server-id=server-b4aa59c4-c124-5230-4930-a189d330efa0 time=78.959142ms testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.806Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to heartbeat to: peer=5ef43523-94a4-651e-1b30-bbc4ced84e32 error="send timed out" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.831Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to pipeline appendEntries: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" error="command enqueue timeout" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.833Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.836Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: pipelining replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testing.go:149: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.929Z [WARN] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: heartbeat timeout reached, starting election: last-leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testing.go:149: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.930Z [WARN] server-b4aa59c4-c124-5230-4930-a189d330efa0: heartbeat timeout reached, starting election: last-leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.930Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: failed to contact: server-id=server-b4aa59c4-c124-5230-4930-a189d330efa0 time=68.459823ms testing.go:149: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.933Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: entering candidate state: node="Node at 5ef43523-94a4-651e-1b30-bbc4ced84e32 [Candidate]" term=3 testing.go:149: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.933Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: entering candidate state: node="Node at b4aa59c4-c124-5230-4930-a189d330efa0 [Candidate]" term=3 testing.go:149: server-b4aa59c4-c124-5230-4930-a189d330efa0: 2021-10-19T21:54:38.934Z [INFO] server-b4aa59c4-c124-5230-4930-a189d330efa0: duplicate requestVote for same term: term=3 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [ERROR] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: peer has newer term, stopping replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-5ef43523-94a4-651e-1b30-bbc4ced84e32 5ef43523-94a4-651e-1b30-bbc4ced84e32}" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.934Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: rejecting vote request since we have a leader: from=5ef43523-94a4-651e-1b30-bbc4ced84e32 leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.935Z [WARN] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: rejecting vote request since we have a leader: from=b4aa59c4-c124-5230-4930-a189d330efa0 leader=11f81a5b-c3cb-a8f3-c73a-953965727a6f testing.go:149: server-5ef43523-94a4-651e-1b30-bbc4ced84e32: 2021-10-19T21:54:38.934Z [INFO] server-5ef43523-94a4-651e-1b30-bbc4ced84e32: duplicate requestVote for same term: term=3 testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.930Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: aborting pipeline replication: peer="{Voter server-b4aa59c4-c124-5230-4930-a189d330efa0 b4aa59c4-c124-5230-4930-a189d330efa0}" testing.go:149: server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: 2021-10-19T21:54:38.935Z [INFO] server-11f81a5b-c3cb-a8f3-c73a-953965727a6f: entering follower state: follower="Node at 11f81a5b-c3cb-a8f3-c73a-953965727a6f [Follower]" leader= raft_test.go:2122: leadership transfer should err with: leadership transfer timeout === RUN TestRaft_LeadershipTransferStopRightAway --- PASS: TestRaft_LeadershipTransferStopRightAway (0.00s) === RUN TestRaft_GetConfigurationNoBootstrap testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:38.964Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:38.964Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering follower state: follower="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Follower]" leader= testLoggerAdapter verbose: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:38.964Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:38.964Z [INFO] starting stability test: raft-state=Follower testLoggerAdapter verbose: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:38.964Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: entering follower state: follower="Node at 34b5b524-306a-d540-9d6a-e6ed6d50e087 [Follower]" leader= testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [WARN] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: heartbeat timeout reached, starting election: last-leader= testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering candidate state: node="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Candidate]" term=2 testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: election won: tally=2 testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering leader state: leader="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Leader]" testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: added peer, starting replication: peer=server-34b5b524-306a-d540-9d6a-e6ed6d50e087 testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.064Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: pipelining replication: peer="{Voter server-34b5b524-306a-d540-9d6a-e6ed6d50e087 34b5b524-306a-d540-9d6a-e6ed6d50e087}" testLoggerAdapter verbose: cluster: 2021-10-19T21:54:39.155Z [INFO] stable state for Follower reached at 2021-10-19 21:54:39.049974584 +0000 UTC m=+20.175000749 (1 nodes), 85.274ms from start of poll, 85.757522ms from cluster start. Timeout at 2021-10-19 21:54:39.155042678 +0000 UTC m=+20.280068832, 105.068083ms after stability testLoggerAdapter verbose: cluster: 2021-10-19T21:54:39.155Z [INFO] starting stability test: raft-state=Leader testLoggerAdapter verbose: cluster: 2021-10-19T21:54:39.260Z [INFO] stable state for Leader reached at 2021-10-19 21:54:39.15526921 +0000 UTC m=+20.280295354 (1 nodes), 2.248µs from start of poll, 191.052127ms from cluster start. Timeout at 2021-10-19 21:54:39.260371279 +0000 UTC m=+20.385397418, 105.102064ms after stability testLoggerAdapter verbose: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:39.270Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testLoggerAdapter verbose: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.271Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: aborting pipeline replication: peer="{Voter server-34b5b524-306a-d540-9d6a-e6ed6d50e087 34b5b524-306a-d540-9d6a-e6ed6d50e087}" --- PASS: TestRaft_GetConfigurationNoBootstrap (0.31s) testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:38.964Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:38.964Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering follower state: follower="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Follower]" leader= testing.go:149: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:38.964Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testing.go:149: cluster: 2021-10-19T21:54:38.964Z [INFO] starting stability test: raft-state=Follower testing.go:149: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:38.964Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: entering follower state: follower="Node at 34b5b524-306a-d540-9d6a-e6ed6d50e087 [Follower]" leader= testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [WARN] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: heartbeat timeout reached, starting election: last-leader= testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering candidate state: node="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Candidate]" term=2 testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: election won: tally=2 testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: entering leader state: leader="Node at 1cbe2b23-0bca-8c49-110d-3828addba4b2 [Leader]" testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.047Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: added peer, starting replication: peer=server-34b5b524-306a-d540-9d6a-e6ed6d50e087 testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.064Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: pipelining replication: peer="{Voter server-34b5b524-306a-d540-9d6a-e6ed6d50e087 34b5b524-306a-d540-9d6a-e6ed6d50e087}" testing.go:149: cluster: 2021-10-19T21:54:39.155Z [INFO] stable state for Follower reached at 2021-10-19 21:54:39.049974584 +0000 UTC m=+20.175000749 (1 nodes), 85.274ms from start of poll, 85.757522ms from cluster start. Timeout at 2021-10-19 21:54:39.155042678 +0000 UTC m=+20.280068832, 105.068083ms after stability testing.go:149: cluster: 2021-10-19T21:54:39.155Z [INFO] starting stability test: raft-state=Leader testing.go:149: cluster: 2021-10-19T21:54:39.260Z [INFO] stable state for Leader reached at 2021-10-19 21:54:39.15526921 +0000 UTC m=+20.280295354 (1 nodes), 2.248µs from start of poll, 191.052127ms from cluster start. Timeout at 2021-10-19 21:54:39.260371279 +0000 UTC m=+20.385397418, 105.102064ms after stability testing.go:149: server-34b5b524-306a-d540-9d6a-e6ed6d50e087: 2021-10-19T21:54:39.270Z [INFO] server-34b5b524-306a-d540-9d6a-e6ed6d50e087: initial configuration: index=1 servers="[{Suffrage:Voter ID:server-1cbe2b23-0bca-8c49-110d-3828addba4b2 Address:1cbe2b23-0bca-8c49-110d-3828addba4b2} {Suffrage:Voter ID:server-34b5b524-306a-d540-9d6a-e6ed6d50e087 Address:34b5b524-306a-d540-9d6a-e6ed6d50e087}]" testing.go:149: server-1cbe2b23-0bca-8c49-110d-3828addba4b2: 2021-10-19T21:54:39.271Z [INFO] server-1cbe2b23-0bca-8c49-110d-3828addba4b2: aborting pipeline replication: peer="{Voter server-34b5b524-306a-d540-9d6a-e6ed6d50e087 34b5b524-306a-d540-9d6a-e6ed6d50e087}" === RUN TestTCPTransport_BadAddr --- PASS: TestTCPTransport_BadAddr (0.00s) === RUN TestTCPTransport_WithAdvertise --- PASS: TestTCPTransport_WithAdvertise (0.00s) === RUN TestTransport_StartStop --- PASS: TestTransport_StartStop (0.00s) === RUN TestTransport_AppendEntries --- PASS: TestTransport_AppendEntries (0.00s) === RUN TestTransport_AppendEntriesPipeline --- PASS: TestTransport_AppendEntriesPipeline (0.00s) === RUN TestTransport_RequestVote --- PASS: TestTransport_RequestVote (0.00s) === RUN TestTransport_InstallSnapshot --- PASS: TestTransport_InstallSnapshot (0.00s) === RUN TestTransport_EncodeDecode --- PASS: TestTransport_EncodeDecode (0.00s) === RUN TestRandomTimeout --- PASS: TestRandomTimeout (0.00s) === RUN TestNewSeed --- PASS: TestNewSeed (0.00s) === RUN TestRandomTimeout_NoTime --- PASS: TestRandomTimeout_NoTime (0.00s) === RUN TestMin --- PASS: TestMin (0.00s) === RUN TestMax --- PASS: TestMax (0.00s) === RUN TestGenerateUUID --- PASS: TestGenerateUUID (0.00s) === RUN TestBackoff --- PASS: TestBackoff (0.00s) FAIL FAIL github.com/hashicorp/raft 20.661s ? github.com/hashicorp/raft/bench [no test files] FAIL dh_auto_test: error: cd obj-x86_64-linux-gnu && go test -vet=off -v -p 4 github.com/hashicorp/raft github.com/hashicorp/raft/bench returned exit code 1 make: *** [debian/rules:9: build-indep] Error 25 dpkg-buildpackage: error: debian/rules build-indep subprocess returned exit status 2 E: run_chroot failed: E: command failed: chroot "$1" env --unset=TMPDIR runuser builduser -c "cd /build/golang-github-hashicorp-raft-HJm3EX/golang-github-hashicorp-raft-1.1.2 && env DEB_BUILD_OPTIONS="parallel=4" LC_ALL="C.UTF-8" SOURCE_DATE_EPOCH="1579418142" dpkg-buildpackage -uc -a amd64 --build=all" W: listening on child socket failed: I: removing tempdir /tmp/mmdebstrap.ZLm5cCJwWc... mmdebstrap failed