Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[13:02:07] Transferring environment variable script to build VM
[13:02:08] Transferring kokoro_log_reader.py to build VM
[13:02:09] Transferring source code to build VM
[13:02:10] Executing build script on build VM



[ID: 7548055] Executing command via SSH:
export KOKORO_BUILD_NUMBER="505"
export KOKORO_JOB_NAME="grpc/node/master/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

Submodule 'packages/grpc-js-xds/deps/envoy-api' (https://github.com/envoyproxy/data-plane-api.git) registered for path '../deps/envoy-api'
Submodule 'packages/grpc-js-xds/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../deps/googleapis'
Submodule 'packages/grpc-js-xds/deps/protoc-gen-validate' (https://github.com/envoyproxy/protoc-gen-validate.git) registered for path '../deps/protoc-gen-validate'
Submodule 'packages/grpc-js-xds/deps/xds' (https://github.com/cncf/xds.git) registered for path '../deps/xds'
Submodule 'packages/grpc-tools/deps/protobuf' (https://github.com/protocolbuffers/protobuf) registered for path '../../grpc-tools/deps/protobuf'
Submodule 'packages/proto-loader/deps/gapic-showcase' (https://github.com/googleapis/gapic-showcase.git) registered for path '../../proto-loader/deps/gapic-showcase'
Submodule 'packages/proto-loader/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../../proto-loader/deps/googleapis'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/envoy-api'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/googleapis'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/protoc-gen-validate'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/xds'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/googleapis'...
Submodule path '../deps/envoy-api': checked out 'e53e7bbd012f81965f2e79848ad9a58ceb67201f'
Submodule path '../deps/googleapis': checked out '8c53b2cb792234354c13336ac7daee61333deade'
Submodule path '../deps/protoc-gen-validate': checked out '0af61d9dc28712dc0e6f8e1a940855a2ee0cb9ed'
Submodule path '../deps/xds': checked out 'cb28da3451f158a947dfc45090fe92b07b243bc1'
Submodule path '../../grpc-tools/deps/protobuf': checked out '7c40b2df1fdf6f414c1c18c789715a9c948a0725'
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path '../../grpc-tools/deps/protobuf/third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path '../../grpc-tools/deps/protobuf/third_party/googletest'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/benchmark'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/googletest'...
Submodule path '../../grpc-tools/deps/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
Submodule path '../../grpc-tools/deps/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
Submodule path '../../proto-loader/deps/gapic-showcase': checked out 'b09b3ba9a8db8aae7d5d7c3939853681cc97c293'
Submodule 'schema/api-common-protos' (https://github.com/googleapis/api-common-protos.git) registered for path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos'
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase/schema/api-common-protos'...
Submodule path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos': checked out '4c0a203e3658ae0e56d47c817c2c5904116c0ae0'
Submodule path '../../proto-loader/deps/googleapis': checked out '8f2eda119e11c8bd0c189b545da18bba9019c83e'
Sourcing test driver install script from: https://raw.githubusercontent.com/murgatroid99/grpc/xds_test_node_enable_affinity/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-node
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
Python 3.10.6
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,1
GIT_ORIGIN_URL,https://github.com/murgatroid99/grpc-node.git
GIT_COMMIT_SHORT,a0262257
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128497 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128442 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128042 files and directories currently installed.)
Preparing to unpack .../00-python3.10-dev_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-dev (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../01-libpython3.10-dev_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-dev:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../02-libpython3.10_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../03-python3.10_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../04-libpython3.10-stdlib_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-stdlib:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../05-python3.10-minimal_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-minimal (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../06-libpython3.10-minimal_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-minimal:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
Preparing to unpack .../07-google-cloud-sdk-gke-gcloud-auth-plugin_446.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (446.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../08-kubectl_1.28.1-00_amd64.deb ...
Unpacking kubectl (1.28.1-00) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../09-python3-pip-whl_22.0.2+dfsg-1ubuntu0.3_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.3) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../10-python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../11-python3.10-venv_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.2) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.3) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (446.0.0-0) ...
Setting up kubectl (1.28.1-00) ...
Setting up libpython3.10-minimal:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10-minimal (3.10.12-1~22.04.2) ...
Setting up libpython3.10-stdlib:amd64 (3.10.12-1~22.04.2) ...
Setting up libpython3.10:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10 (3.10.12-1~22.04.2) ...
Setting up libpython3.10-dev:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10-dev (3.10.12-1~22.04.2) ...
Setting up python3.10-venv (3.10.12-1~22.04.2) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/murgatroid99/grpc.git branch xds_test_node_enable_affinity to /tmp/tmp.WKlxHh40Vl/grpc
Cloning into '/tmp/tmp.WKlxHh40Vl/grpc'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 1.9 MB/s eta 0:00:00
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 7.7 MB/s eta 0:00:00
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 8.4 MB/s eta 0:00:00
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 10.0 MB/s eta 0:00:00
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 9.8 MB/s eta 0:00:00
Collecting grpcio==1.57.0 (from -r requirements.lock (line 6))
Obtaining dependency information for grpcio==1.57.0 from https://files.pythonhosted.org/packages/28/fa/c38a010d3fffcac07ef121abb34eb2c3db0876df74267ce5bde13c3a6ed7/grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting grpcio-health-checking==1.57.0 (from -r requirements.lock (line 7))
Obtaining dependency information for grpcio-health-checking==1.57.0 from https://files.pythonhosted.org/packages/03/0b/fc420732783c747a099ae9d41c8fd728cbe9751859532330c6bde18f7874/grpcio_health_checking-1.57.0-py3-none-any.whl.metadata
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting grpcio-tools==1.57.0 (from -r requirements.lock (line 8))
Obtaining dependency information for grpcio-tools==1.57.0 from https://files.pythonhosted.org/packages/00/8b/903898fcaf31f67180061983f6bf54027bc85ffb6f7b202f8263388fc2ae/grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio-channelz==1.57.0 (from -r requirements.lock (line 9))
Obtaining dependency information for grpcio-channelz==1.57.0 from https://files.pythonhosted.org/packages/0f/0e/1fd7c577032d68933c89b8a0cec0b26543c31625c1a19e6151dead14a520/grpcio_channelz-1.57.0-py3-none-any.whl.metadata
Downloading grpcio_channelz-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 10))
Obtaining dependency information for kubernetes==27.2.0 from https://files.pythonhosted.org/packages/99/89/3ab0cb3069f49ae2eaf73f884c82164f18f70fcc598e0312edea71614ad7/kubernetes-27.2.0-py2.py3-none-any.whl.metadata
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 11))
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 12))
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==23.1 (from -r requirements.lock (line 13))
Downloading packaging-23.1-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 9.7 MB/s eta 0:00:00
Collecting Pygments==2.14.0 (from -r requirements.lock (line 14))
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 11.3 MB/s eta 0:00:00
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 15))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 11.7 MB/s eta 0:00:00
Collecting protobuf==4.24.1 (from -r requirements.lock (line 16))
Obtaining dependency information for protobuf==4.24.1 from https://files.pythonhosted.org/packages/4c/87/59648989ad7f5ba6fe3c7f8abc555183f28559b6f6cd14ad17a3f0d3094f/protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting xds-protos==1.58.0rc1 (from -r requirements.lock (line 17))
Obtaining dependency information for xds-protos==1.58.0rc1 from https://files.pythonhosted.org/packages/4d/70/36181d7ba0f09495a61f46ce6cdec95eb6224c2f02cd984c4dcf9977b4d0/xds_protos-1.58.0rc1-py3-none-any.whl.metadata
Downloading xds_protos-1.58.0rc1-py3-none-any.whl.metadata (1.3 kB)
Collecting cachetools==5.3.1 (from -r requirements.lock (line 19))
Obtaining dependency information for cachetools==5.3.1 from https://files.pythonhosted.org/packages/a9/c9/c8a7710f2cedcb1db9224fdd4d8307c9e48cbddc46c18b515fefc0f1abbe/cachetools-5.3.1-py3-none-any.whl.metadata
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.lock (line 20))
Obtaining dependency information for certifi==2023.7.22 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.2.0 (from -r requirements.lock (line 21))
Obtaining dependency information for charset-normalizer==3.2.0 from https://files.pythonhosted.org/packages/a4/65/057bf29660aae6ade0816457f8db4e749e5c0bfa2366eb5f67db9912fa4c/charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting google-api-core==2.11.1 (from -r requirements.lock (line 22))
Obtaining dependency information for google-api-core==2.11.1 from https://files.pythonhosted.org/packages/6e/c4/c3cd048b6cbeba8d9ae50dd7643ac065b85237338aa7501b0efae91eb4d9/google_api_core-2.11.1-py3-none-any.whl.metadata
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0 (from -r requirements.lock (line 23))
Obtaining dependency information for google-auth==2.22.0 from https://files.pythonhosted.org/packages/9c/8d/bff87fc722553a5691d8514da5523c23547f3894189ba03b57592e37bdc2/google_auth-2.22.0-py2.py3-none-any.whl.metadata
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 24))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.60.0 (from -r requirements.lock (line 25))
Obtaining dependency information for googleapis-common-protos==1.60.0 from https://files.pythonhosted.org/packages/a7/bc/416a1ffeba4dcd072bc10523dac9ed97f2e7fc4b760580e2bdbdc1e2afdd/googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 26))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.57.0 (from -r requirements.lock (line 27))
Obtaining dependency information for grpcio-status==1.57.0 from https://files.pythonhosted.org/packages/d0/3f/347d93056572fdbd64d4f0fc58a18d420763a7118f8b177437d9dab0ae6f/grpcio_status-1.57.0-py3-none-any.whl.metadata
Downloading grpcio_status-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting httplib2==0.22.0 (from -r requirements.lock (line 28))
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 13.6 MB/s eta 0:00:00
Collecting idna==3.4 (from -r requirements.lock (line 29))
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 11.1 MB/s eta 0:00:00
Collecting MarkupSafe==2.1.3 (from -r requirements.lock (line 30))
Obtaining dependency information for MarkupSafe==2.1.3 from https://files.pythonhosted.org/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 31))
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 14.3 MB/s eta 0:00:00
Collecting proto-plus==1.22.3 (from -r requirements.lock (line 32))
Obtaining dependency information for proto-plus==1.22.3 from https://files.pythonhosted.org/packages/36/5b/e02636d221917d6fa2a61289b3f16002eb4c93d51c0191ac8e896d527182/proto_plus-1.22.3-py3-none-any.whl.metadata
Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.lock (line 33))
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 12.9 MB/s eta 0:00:00
Collecting pyasn1-modules==0.3.0 (from -r requirements.lock (line 34))
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 14.8 MB/s eta 0:00:00
Collecting pyparsing==3.1.1 (from -r requirements.lock (line 35))
Obtaining dependency information for pyparsing==3.1.1 from https://files.pythonhosted.org/packages/39/92/8486ede85fcc088f1b3dba4ce92dd29d126fd96b0008ea213167940a2475/pyparsing-3.1.1-py3-none-any.whl.metadata
Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting requests==2.31.0 (from -r requirements.lock (line 36))
Obtaining dependency information for requests==2.31.0 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 37))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 38))
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 39))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.16 (from -r requirements.lock (line 40))
Obtaining dependency information for urllib3==1.26.16 from https://files.pythonhosted.org/packages/c5/05/c214b32d21c0b465506f95c4f28ccbcba15022e000b043b72b3df7728471/urllib3-1.26.16-py2.py3-none-any.whl.metadata
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 8.3 MB/s eta 0:00:00
Collecting websocket-client==1.6.2 (from -r requirements.lock (line 41))
Obtaining dependency information for websocket-client==1.6.2 from https://files.pythonhosted.org/packages/4b/4a/3176388095e5bae6e6a1fbee66c438809230ae0196e7de4af12c5e75c509/websocket_client-1.6.2-py3-none-any.whl.metadata
Downloading websocket_client-1.6.2-py3-none-any.whl.metadata (7.5 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.57.0->-r requirements.lock (line 8)) (68.2.2)
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 21.2 MB/s eta 0:00:00
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl (8.6 kB)
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.9 MB/s eta 0:00:00
Downloading grpcio_channelz-1.57.0-py3-none-any.whl (10 kB)
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 15.9 MB/s eta 0:00:00
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.4/311.4 kB 34.9 MB/s eta 0:00:00
Downloading xds_protos-1.58.0rc1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 40.2 MB/s eta 0:00:00
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 26.8 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.8/201.8 kB 32.9 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 22.4 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 27.1 MB/s eta 0:00:00
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 33.9 MB/s eta 0:00:00
Downloading grpcio_status-1.57.0-py3-none-any.whl (5.1 kB)
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 7.2 MB/s eta 0:00:00
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 17.7 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 11.3 MB/s eta 0:00:00
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 24.1 MB/s eta 0:00:00
Downloading websocket_client-1.6.2-py3-none-any.whl (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 11.3 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, uritemplate, six, PyYAML, pyparsing, Pygments, pyasn1, protobuf, packaging, oauthlib, MarkupSafe, idna, grpcio, charset-normalizer, certifi, cachetools, xds-protos, tenacity, rsa, requests, python-dateutil, pyasn1-modules, proto-plus, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, absl-py, requests-oauthlib, grpcio-status, google-auth, kubernetes, grpc-google-iam-v1, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-secret-manager
Successfully installed Mako-1.2.4 MarkupSafe-2.1.3 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-api-core-2.11.1 google-api-python-client-1.12.11 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.57.0 grpcio-channelz-1.57.0 grpcio-health-checking-1.57.0 grpcio-status-1.57.0 grpcio-tools-1.57.0 httplib2-0.22.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.1 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.16 websocket-client-1.6.2 xds-protos-1.58.0rc1
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.1
certifi 2023.7.22
charset-normalizer 3.2.0
google-api-core 2.11.1
google-api-python-client 1.12.11
google-auth 2.22.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.60.0
grpc-google-iam-v1 0.12.6
grpcio 1.57.0
grpcio-channelz 1.57.0
grpcio-health-checking 1.57.0
grpcio-status 1.57.0
grpcio-tools 1.57.0
httplib2 0.22.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.3
oauthlib 3.2.2
packaging 23.1
pip 23.2.1
proto-plus 1.22.3
protobuf 4.24.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
Pygments 2.14.0
pyparsing 3.1.1
python-dateutil 2.8.2
PyYAML 6.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 68.2.2
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.16
websocket-client 1.6.2
xds-protos 1.58.0rc1
Generate python code from grpc.testing protos: src/proto/grpc/testing/test.proto src/proto/grpc/testing/messages.proto src/proto/grpc/testing/empty.proto
/tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing:
total 80
-rw-rw-r-- 1 kbuilder kbuilder 1014 Sep 12 20:03 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Sep 12 20:03 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 16959 Sep 12 20:03 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Sep 12 20:03 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 4139 Sep 12 20:03 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 39295 Sep 12 20:03 test_pb2_grpc.py
Client image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
Client image not found
Building Node xDS interop test app Docker images
/tmpfs/src/github/grpc-node /tmp/tmp.WKlxHh40Vl/grpc
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.72kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/node:18-slim
#3 ...

#4 [internal] load metadata for gcr.io/distroless/nodejs18-debian11:latest
#4 DONE 0.4s

#3 [internal] load metadata for docker.io/library/node:18-slim
#3 DONE 0.6s

#5 [internal] load build context
#5 DONE 0.0s

#6 [build 1/7] FROM docker.io/library/node:18-slim@sha256:e5c8c319295f6cbc288e19506a9ac37afa3b330f4e38afb01d1269b579cf6a5b
#6 resolve docker.io/library/node:18-slim@sha256:e5c8c319295f6cbc288e19506a9ac37afa3b330f4e38afb01d1269b579cf6a5b 0.0s done
#6 sha256:e5c8c319295f6cbc288e19506a9ac37afa3b330f4e38afb01d1269b579cf6a5b 1.21kB / 1.21kB done
#6 sha256:abe13b25e07ccaedcc8797120c37781f0462a0f0682105cf2d8b8d6f99070d55 1.37kB / 1.37kB done
#6 sha256:4c51186e5354597a05e622cb3cc53bf18e8c9898a84d641f64d615e588fbac52 7.02kB / 7.02kB done
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 0B / 29.12MB 0.5s
#6 sha256:24f632c8bcc8f838302a408474f63c842f77e774d05328594fb0e195569cd7aa 0B / 3.36kB 0.5s
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 5.24MB / 29.12MB 0.6s
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 15.73MB / 29.12MB 0.8s
#6 sha256:24f632c8bcc8f838302a408474f63c842f77e774d05328594fb0e195569cd7aa 3.36kB / 3.36kB 0.6s done
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 4.19MB / 46.24MB 0.8s
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 24.12MB / 29.12MB 0.9s
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 11.53MB / 46.24MB 0.9s
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 29.12MB / 29.12MB 1.0s
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 24.12MB / 46.24MB 1.0s
#6 sha256:e03d23b55cf6a48bc1cf4382e9873dd1c27ea08a042c6dd2ff1ab4b740b8c70c 0B / 2.74MB 1.0s
#6 sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 29.12MB / 29.12MB 1.0s done
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 34.60MB / 46.24MB 1.1s
#6 sha256:e03d23b55cf6a48bc1cf4382e9873dd1c27ea08a042c6dd2ff1ab4b740b8c70c 2.74MB / 2.74MB 1.0s done
#6 extracting sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0
#6 sha256:e65223f9c4818b13c2bcc9ea7550f9c26380a9822962a294fc50c27c01ef508a 0B / 453B 1.1s
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 46.24MB / 46.24MB 1.2s
#6 sha256:e65223f9c4818b13c2bcc9ea7550f9c26380a9822962a294fc50c27c01ef508a 453B / 453B 1.1s done
#6 sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 46.24MB / 46.24MB 1.2s done
#6 extracting sha256:360eba32fa65016e0d558c6af176db31a202e9a6071666f9b629cb8ba6ccedf0 1.1s done
#6 extracting sha256:24f632c8bcc8f838302a408474f63c842f77e774d05328594fb0e195569cd7aa done
#6 extracting sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971
#6 ...

#7 [stage-1 1/4] FROM gcr.io/distroless/nodejs18-debian11:latest@sha256:117e714f608555028a18c8162db6246557ec667159d18714a4dd7a9ee5948be2
#7 resolve gcr.io/distroless/nodejs18-debian11:latest@sha256:117e714f608555028a18c8162db6246557ec667159d18714a4dd7a9ee5948be2 0.0s done
#7 extracting sha256:a7ca0d9ba68fdce7e15bc0952d3e898e970548ca24d57698725836c039086639 done
#7 sha256:fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265 317B / 317B 0.2s done
#7 sha256:a7ca0d9ba68fdce7e15bc0952d3e898e970548ca24d57698725836c039086639 103.73kB / 103.73kB 0.1s done
#7 sha256:a345696a4985d7427e9aa0d22323a1cc5d3eba6abd77e792c48cc5fb21aa02b7 2.07kB / 2.07kB done
#7 sha256:b02a7525f878e61fc1ef8a7405a2cc17f866e8de222c1c98fd6681aff6e509db 716.49kB / 716.49kB 0.1s done
#7 sha256:2b5a7f28895b399a6d16b057a8ca5d72a5add897425ecab66b5202d821c0b641 2.75kB / 2.75kB done
#7 sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58 21.20kB / 21.20kB 0.1s done
#7 sha256:117e714f608555028a18c8162db6246557ec667159d18714a4dd7a9ee5948be2 673B / 673B done
#7 extracting sha256:fe5ca62666f04366c8e7f605aa82997d71320183e99962fa76b3209fdfbb8b58 done
#7 sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f 385B / 385B 0.2s done
#7 sha256:7c881f9ab25e0d86562a123b5fb56aebf8aa0ddd7d48ef602faf8d1e7cf43d8c 355B / 355B 0.2s done
#7 sha256:5627a970d25e752d971a501ec7e35d0d6fdcd4a3ce9e958715a686853024794a 130.56kB / 130.56kB 0.2s done
#7 sha256:e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0 198B / 198B 0.1s done
#7 sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c 113B / 113B 0.2s done
#7 extracting sha256:b02a7525f878e61fc1ef8a7405a2cc17f866e8de222c1c98fd6681aff6e509db 0.2s done
#7 sha256:96266735468f361ae6828901a80fc15a7f75e26640351df9e0f0f9824f36cf92 5.85MB / 5.85MB 0.4s done
#7 sha256:2758d0c31c8ca76c3379e7b1be20adc4144e9230873bb2c5bdb41f3691fa75bc 2.06MB / 2.06MB 0.3s done
#7 sha256:08553ba93cfea7ad45b59911d8ed0a025489e7c3623920dfda331b9a49f1e8aa 961.77kB / 961.77kB 0.3s done
#7 sha256:4eec690774a46467a912715848c71dbbdb049008b2252432155522a7f9ccfa92 52.43kB / 52.43kB 0.4s done
#7 sha256:dfc02eb7708f919bb3b56c008561e4430ea87cd33bc93cb65c2c3c7f0908e5cf 131.16kB / 131.16kB 0.4s done
#7 sha256:52907d314ddce378f3f36e26629baef60c71d72a0620b9d31c47c8cb9de6467e 741.34kB / 741.34kB 0.4s done
#7 sha256:77f03d261637800e69f631d0d061d7cc3216ac542af0bc4117468961705af79e 50.01MB / 50.01MB 0.9s done
#7 extracting sha256:fcb6f6d2c9986d9cd6a2ea3cc2936e5fc613e09f1af9042329011e43057f3265 done
#7 extracting sha256:e8c73c638ae9ec5ad70c49df7e484040d889cca6b4a9af056579c3d058ea93f0 done
#7 extracting sha256:1e3d9b7d145208fa8fa3ee1c9612d0adaac7255f1bbc9ddea7e461e0b317805c done
#7 extracting sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f done
#7 extracting sha256:7c881f9ab25e0d86562a123b5fb56aebf8aa0ddd7d48ef602faf8d1e7cf43d8c done
#7 extracting sha256:5627a970d25e752d971a501ec7e35d0d6fdcd4a3ce9e958715a686853024794a done
#7 extracting sha256:96266735468f361ae6828901a80fc15a7f75e26640351df9e0f0f9824f36cf92 0.2s done
#7 extracting sha256:2758d0c31c8ca76c3379e7b1be20adc4144e9230873bb2c5bdb41f3691fa75bc 0.1s done
#7 extracting sha256:08553ba93cfea7ad45b59911d8ed0a025489e7c3623920dfda331b9a49f1e8aa 0.1s done
#7 extracting sha256:dfc02eb7708f919bb3b56c008561e4430ea87cd33bc93cb65c2c3c7f0908e5cf done
#7 extracting sha256:52907d314ddce378f3f36e26629baef60c71d72a0620b9d31c47c8cb9de6467e 0.1s done
#7 extracting sha256:4eec690774a46467a912715848c71dbbdb049008b2252432155522a7f9ccfa92 done
#7 extracting sha256:77f03d261637800e69f631d0d061d7cc3216ac542af0bc4117468961705af79e 1.5s done
#7 DONE 4.2s

#6 [build 1/7] FROM docker.io/library/node:18-slim@sha256:e5c8c319295f6cbc288e19506a9ac37afa3b330f4e38afb01d1269b579cf6a5b
#6 extracting sha256:79a793b37a27577c228b17f66ddcbc1d1e500d71ac167ef4b99f0e81adadf971 1.9s done
#6 extracting sha256:e03d23b55cf6a48bc1cf4382e9873dd1c27ea08a042c6dd2ff1ab4b740b8c70c
#6 extracting sha256:e03d23b55cf6a48bc1cf4382e9873dd1c27ea08a042c6dd2ff1ab4b740b8c70c 0.2s done
#6 extracting sha256:e65223f9c4818b13c2bcc9ea7550f9c26380a9822962a294fc50c27c01ef508a done
#6 DONE 6.6s

#8 [build 2/7] WORKDIR /node/src/grpc-node
#8 ...

#5 [internal] load build context
#5 transferring context: 702.15MB 7.2s done
#5 DONE 7.5s

#9 [stage-1 2/4] WORKDIR /node/src/grpc-node
#9 DONE 3.7s

#8 [build 2/7] WORKDIR /node/src/grpc-node
#8 DONE 1.3s

#10 [build 3/7] COPY . .
#10 DONE 4.4s

#11 [build 4/7] WORKDIR /node/src/grpc-node/packages/grpc-js
#11 DONE 0.0s

#12 [build 5/7] RUN npm install
#12 22.48 npm WARN deprecated flatten@1.0.3: flatten is deprecated in favor of utility frameworks such as lodash.
#12 22.57 npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
#12 22.57 npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
#12 22.64 npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
#12 22.73 npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
#12 25.96 npm WARN deprecated chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
#12 26.08 npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
#12 28.95
#12 28.95 > @grpc/grpc-js@1.8.21 prepare
#12 28.95 > npm run generate-types && npm run compile
#12 28.95
#12 29.29
#12 29.29 > @grpc/grpc-js@1.8.21 generate-types
#12 29.29 > proto-loader-gen-types --keepCase --longs String --enums String --defaults --oneofs --includeComments --includeDirs proto/ --include-dirs test/fixtures/ -O src/generated/ --grpcLib ../index channelz.proto
#12 29.29
#12 29.84
#12 29.84 > @grpc/grpc-js@1.8.21 compile
#12 29.84 > tsc -p .
#12 29.84
#12 35.80
#12 35.80 added 733 packages, and audited 734 packages in 35s
#12 35.80
#12 35.80 76 packages are looking for funding
#12 35.80 run `npm fund` for details
#12 35.81
#12 35.81 11 vulnerabilities (8 high, 3 critical)
#12 35.81
#12 35.81 To address issues that do not require attention, run:
#12 35.81 npm audit fix
#12 35.81
#12 35.81 To address all issues (including breaking changes), run:
#12 35.81 npm audit fix --force
#12 35.81
#12 35.81 Run `npm audit` for details.
#12 35.81 npm notice
#12 35.81 npm notice New major version of npm available! 9.6.7 -> 10.1.0
#12 35.81 npm notice Changelog:
#12 35.81 npm notice Run `npm install -g npm@10.1.0` to update!
#12 35.81 npm notice
#12 DONE 36.7s

#13 [build 6/7] WORKDIR /node/src/grpc-node/packages/grpc-js-xds
#13 DONE 0.0s

#14 [build 7/7] RUN npm install
#14 17.90
#14 17.90 > @grpc/grpc-js-xds@1.8.2 prepare
#14 17.90 > npm run compile
#14 17.90
#14 18.23
#14 18.23 > @grpc/grpc-js-xds@1.8.2 compile
#14 18.23 > tsc
#14 18.23
#14 24.67
#14 24.67 added 364 packages, and audited 366 packages in 24s
#14 24.67
#14 24.67 79 packages are looking for funding
#14 24.67 run `npm fund` for details
#14 24.68
#14 24.68 found 0 vulnerabilities
#14 DONE 25.3s

#15 [stage-1 3/4] COPY --from=build /node/src/grpc-node/packages/grpc-js ./packages/grpc-js/
#15 DONE 2.7s

#16 [stage-1 4/4] COPY --from=build /node/src/grpc-node/packages/grpc-js-xds ./packages/grpc-js-xds/
#16 DONE 2.8s

#17 exporting to image
#17 exporting layers
#17 exporting layers 2.7s done
#17 writing image sha256:49f1988dff889eaa8aeceb3f593e53ff9012675968bd7db42b9a29f6d10b70fb done
#17 naming to gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0 done
#17 DONE 2.7s
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
The push refers to repository [gcr.io/grpc-testing/xds-interop/node-client]
d602206f73fb: Preparing
be82d2073d9e: Preparing
7d7e9ca516ef: Preparing
55421bbff262: Preparing
5bf213caca44: Preparing
dcecd3d0367d: Preparing
bb0331ba4692: Preparing
6a1069d9378c: Preparing
1c47a89b8f41: Preparing
c60b28d3f33c: Preparing
4cb10dd2545b: Preparing
d2d7ec0f6756: Preparing
1a73b54f556b: Preparing
e624a5370eca: Preparing
d52f02c6501c: Preparing
ff5700ec5418: Preparing
7bea6b893187: Preparing
6fbdf253bbc2: Preparing
e023e0e48e6e: Preparing
6a1069d9378c: Waiting
1c47a89b8f41: Waiting
c60b28d3f33c: Waiting
4cb10dd2545b: Waiting
e023e0e48e6e: Waiting
7bea6b893187: Waiting
1a73b54f556b: Waiting
d52f02c6501c: Waiting
e624a5370eca: Waiting
ff5700ec5418: Waiting
6fbdf253bbc2: Waiting
dcecd3d0367d: Waiting
bb0331ba4692: Waiting
d2d7ec0f6756: Waiting
55421bbff262: Layer already exists
5bf213caca44: Layer already exists
dcecd3d0367d: Layer already exists
bb0331ba4692: Layer already exists
6a1069d9378c: Layer already exists
1c47a89b8f41: Layer already exists
c60b28d3f33c: Layer already exists
4cb10dd2545b: Layer already exists
d2d7ec0f6756: Layer already exists
1a73b54f556b: Layer already exists
e624a5370eca: Layer already exists
d52f02c6501c: Layer already exists
ff5700ec5418: Layer already exists
7bea6b893187: Layer already exists
6fbdf253bbc2: Layer already exists
e023e0e48e6e: Layer already exists
7d7e9ca516ef: Pushed
d602206f73fb: Pushed
be82d2073d9e: Pushed
a02622572aec2b236257b86842d26eb32e5805f0: digest: sha256:1c352ea6c779e6befb3b07d39001c5465e58b5d57aabf3d8e499ba68c3b09465 size: 4295
/tmp/tmp.WKlxHh40Vl/grpc
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/affinity_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:05:39.641585 139691887030272 xds_k8s_testcase.py:155] ----- Testing AffinityTest -----
I0912 20:05:39.641733 139691887030272 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:05:39.641990 139691887030272 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:05:40.624998 139691887030272 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:05:40.805333 139691887030272 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0912 20:05:41.098123 139691887030272 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2005-h57dy
I0912 20:05:41.553048 139691887030272 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/affinity_test
I0912 20:05:41.966615 139691887030272 xds_k8s_testcase.py:648] Found unused xds port: 47704
I0912 20:05:41.967053 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0912 20:05:41.967181 139691887030272 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2005-h57dy"
I0912 20:05:41.970362 139691887030272 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2005-h57dy
type: GRPC
...

I0912 20:05:42.777417 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549142044-6052ef7aa803b-230f977b-aceb2f12
I0912 20:05:45.104845 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0912 20:05:45.105058 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0912 20:05:45.105151 139691887030272 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2005-h57dy"
I0912 20:05:45.113024 139691887030272 compute.py:530] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2005-h57dy
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20230912-2005-h57dy
protocol: GRPC
sessionAffinity: HEADER_FIELD
...

I0912 20:05:45.817108 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549145115-6052ef7d95c43-a90a1aef-702ec30b
I0912 20:06:05.604317 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0912 20:06:05.604550 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0912 20:06:05.604648 139691887030272 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2005-h57dy": psm-grpc-server:47704 -> psm-interop-backend-service-20230912-2005-h57dy
I0912 20:06:05.615975 139691887030272 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2005-h57dy
hostRules:
- hosts:
- psm-grpc-server:47704
pathMatcher: psm-interop-path-matcher-20230912-2005-h57dy
name: psm-interop-url-map-20230912-2005-h57dy
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2005-h57dy
name: psm-interop-path-matcher-20230912-2005-h57dy
...

I0912 20:06:06.316981 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549165618-6052ef9123548-10d7241d-664cad76
I0912 20:06:08.584437 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0912 20:06:08.584657 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0912 20:06:08.584793 139691887030272 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2005-h57dy proxy "GRPC" to URL map psm-interop-url-map-20230912-2005-h57dy
I0912 20:06:08.586746 139691887030272 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2005-h57dy
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2005-h57dy
validate_for_proxyless: true
...

I0912 20:06:09.403823 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549168588-6052ef93f87d0-cbd0860d-6465ca28
I0912 20:06:11.788789 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0912 20:06:11.789028 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0912 20:06:11.789135 139691887030272 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2005-h57dy" in network "default-vpc": 0.0.0.0:47704 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2005-h57dy
I0912 20:06:11.791761 139691887030272 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2005-h57dy
network: global/networks/default-vpc
portRange: 47704
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2005-h57dy
...

I0912 20:06:12.533204 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549171794-6052ef970729d-1974deef-3032d321
I0912 20:06:36.537754 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0912 20:06:36.537991 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0912 20:06:36.538127 139691887030272 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2005-h57dy: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0912 20:06:36.538384 139691887030272 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2005-h57dy%22;timeRange=2023-09-12T20:06:36.538213Z%2F2023-09-12T20:36:36.538213Z?project=grpc-testing
I0912 20:06:36.543658 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2005-h57dy
labels:
name: psm-interop-server-20230912-2005-h57dy
owner: xds-k8s-interop-test
...

I0912 20:06:36.606955 139691887030272 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2005-h57dy created
I0912 20:06:36.609930 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2005-h57dy
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2005-h57dy-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:06:36.673798 139691887030272 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:06:36.674023 139691887030272 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:06:36.770173 139691887030272 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2005-h57dy-psm-grpc-server in zones=['us-central1-a']
I0912 20:06:36.770386 139691887030272 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2005-h57dy/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:06:37.519065 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2005-h57dy
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:06:37.581275 139691887030272 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:06:37.589602 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2005-h57dy
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5pop2vcc
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5pop2vcc
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5pop2vcc
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2005-h57dy-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:06:37.663552 139691887030272 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:06:37.663800 139691887030272 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0912 20:06:37.823593 139691887030272 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-68877867c4-hfc66', 'psm-grpc-server-68877867c4-hrrv4', 'psm-grpc-server-68877867c4-wr5tg']
I0912 20:06:37.824012 139691887030272 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-68877867c4-hfc66 to start
I0912 20:06:40.021069 139691887030272 k8s_base_runner.py:628] Pod psm-grpc-server-68877867c4-hfc66 ready, IP: 10.120.13.203
I0912 20:06:40.021341 139691887030272 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-68877867c4-hfc66 to affinity_test/test_app_logs/psm-interop-server-20230912-2005-h57dy_psm-grpc-server-68877867c4-hfc66.log
I0912 20:06:40.021831 139691796457024 k8s_log_collector.py:70] Starting log collection thread 139691796457024 for psm-grpc-server-68877867c4-hfc66
I0912 20:06:40.021985 139691887030272 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-68877867c4-hrrv4 to start
I0912 20:06:40.204047 139691887030272 k8s_base_runner.py:628] Pod psm-grpc-server-68877867c4-hrrv4 ready, IP: 10.120.17.247
I0912 20:06:40.204311 139691887030272 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-68877867c4-hrrv4 to affinity_test/test_app_logs/psm-interop-server-20230912-2005-h57dy_psm-grpc-server-68877867c4-hrrv4.log
I0912 20:06:40.204783 139691788064320 k8s_log_collector.py:70] Starting log collection thread 139691788064320 for psm-grpc-server-68877867c4-hrrv4
I0912 20:06:40.204937 139691887030272 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-68877867c4-wr5tg to start
I0912 20:06:40.444080 139691887030272 k8s_base_runner.py:628] Pod psm-grpc-server-68877867c4-wr5tg ready, IP: 10.120.9.131
I0912 20:06:40.444347 139691887030272 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-68877867c4-wr5tg to affinity_test/test_app_logs/psm-interop-server-20230912-2005-h57dy_psm-grpc-server-68877867c4-wr5tg.log
I0912 20:06:40.444783 139691779671616 k8s_log_collector.py:70] Starting log collection thread 139691779671616 for psm-grpc-server-68877867c4-wr5tg
I0912 20:06:40.444925 139691887030272 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0912 20:06:48.932425 139691887030272 k8s_base_runner.py:596] Deployment psm-grpc-server has 3 replicas available
I0912 20:06:48.933079 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0912 20:06:48.933212 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0912 20:06:48.969306 139691887030272 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:06:49.253513 139691887030272 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2005-h57dy-psm-grpc-server" in zone us-central1-a
I0912 20:06:49.253761 139691887030272 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2005-h57dy: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2005-h57dy-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2005-h57dy-psm-grpc-server', zone='us-central1-a')}
I0912 20:06:49.257766 139691887030272 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2005-h57dy-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 20:06:50.163927 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549209259-6052efbac1fdf-89bb15c5-93e96059
I0912 20:07:50.916591 139691887030272 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2005-h57dy to report all backends healthy: ['psm-interop-server-20230912-2005-h57dy-psm-grpc-server']
I0912 20:08:07.940394 139691887030272 compute.py:487] Backend psm-interop-server-20230912-2005-h57dy-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:08:11.945069 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0912 20:08:11.945320 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0912 20:08:11.945436 139691887030272 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2005-h57dy: server_target=xds:///psm-grpc-server:47704 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0912 20:08:11.945669 139691887030272 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2005-h57dy%22;timeRange=2023-09-12T20:08:11.945519Z%2F2023-09-12T20:38:11.945519Z?project=grpc-testing
I0912 20:08:11.947737 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2005-h57dy
labels:
name: psm-interop-client-20230912-2005-h57dy
owner: xds-k8s-interop-test
...

I0912 20:08:12.028582 139691887030272 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2005-h57dy created
I0912 20:08:12.028861 139691887030272 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2005-h57dy/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:08:12.790393 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2005-h57dy
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:08:12.853133 139691887030272 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:08:12.861356 139691887030272 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2005-h57dy
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jg7abuwt
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jg7abuwt
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jg7abuwt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:47704"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:08:12.936793 139691887030272 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:08:12.937044 139691887030272 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:08:13.039196 139691887030272 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-54f9d6dc9-nttrs']
I0912 20:08:13.039489 139691887030272 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-54f9d6dc9-nttrs to start
I0912 20:08:30.870320 139691887030272 k8s_base_runner.py:628] Pod psm-grpc-client-54f9d6dc9-nttrs ready, IP: 10.120.10.208
I0912 20:08:30.870627 139691887030272 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-54f9d6dc9-nttrs to affinity_test/test_app_logs/psm-interop-client-20230912-2005-h57dy_psm-grpc-client-54f9d6dc9-nttrs.log
I0912 20:08:30.871036 139691771278912 k8s_log_collector.py:70] Starting log collection thread 139691771278912 for psm-grpc-client-54f9d6dc9-nttrs
I0912 20:08:30.871183 139691887030272 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:08:35.211282 139691887030272 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:08:35.211873 139691887030272 client_app.py:200] [psm-grpc-client-54f9d6dc9-nttrs] Waiting to report a READY channel to xds:///psm-grpc-server:47704
I0912 20:08:35.342826 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:08:45.428544 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:08:55.509632 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:09:05.591464 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:09:15.673415 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:09:31.762181 139691887030272 client_app.py:236] [psm-grpc-client-54f9d6dc9-nttrs] Server channel:
I0912 20:09:31.799460 139691887030272 client_app.py:249] [psm-grpc-client-54f9d6dc9-nttrs] Found subchannel in state READY:
I0912 20:09:31.799764 139691887030272 client_app.py:211] [psm-grpc-client-54f9d6dc9-nttrs] Channel to xds:///psm-grpc-server:47704 transitioned to state READY:
I0912 20:09:31.800015 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:09:31.847689 139691887030272 affinity_test.py:108] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/e5f03932-b5fa-4afd-ad1c-61d14fa5b706",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.10.208",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.8.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:47704",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:47704",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-09-12T20:09:28.141Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:47704"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704-route-0"
}
]
}
]
},
"lastUpdated": "2023-09-12T20:09:28.178Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230912-2005-h57dy"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230912-2005-h57dy",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-09-12T20:09:28.220Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"versionInfo": "1",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3806992279121263138_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.247",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.9.131",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.13.203",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-09-12T20:09:29.634Z",
"clientStatus": "ACKED"
}
]
}
I0912 20:09:31.848498 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0912 20:09:31.848633 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0912 20:09:31.848790 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:09:31.884534 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0912 20:09:31.884782 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0912 20:09:31.885003 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:09:35.925694 139691887030272 xds_k8s_testcase.py:586] [psm-grpc-client-54f9d6dc9-nttrs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-68877867c4-hfc66: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-68877867c4-hfc66: 100

I0912 20:09:35.925992 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0912 20:09:35.926158 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0912 20:09:35.926346 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:09:40.019269 139691887030272 xds_k8s_testcase.py:586] [psm-grpc-client-54f9d6dc9-nttrs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-68877867c4-hfc66: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-68877867c4-hfc66: 100

I0912 20:09:40.124801 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0912 20:09:40.125020 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0912 20:09:40.125115 139691887030272 server_app.py:98] [psm-grpc-server-68877867c4-hfc66] >> Setting health status to NOT_SERVING
I0912 20:09:40.125470 139691887030272 grpc.py:74] [psm-grpc-server-68877867c4-hfc66:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 20:09:40.225147 139691887030272 grpc.py:74] [psm-grpc-server-68877867c4-hfc66:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 20:09:40.258388 139691887030272 server_app.py:102] [psm-grpc-server-68877867c4-hfc66] << Health status status: NOT_SERVING

I0912 20:09:40.258727 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0912 20:09:40.258849 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0912 20:09:40.258990 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:09:40.294699 139691887030272 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0912 20:09:50.301335 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:09:50.338460 139691887030272 affinity_test.py:185] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/e5f03932-b5fa-4afd-ad1c-61d14fa5b706",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.10.208",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.8.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:47704",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:47704",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-09-12T20:09:28.141Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:47704"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2005-h57dy_0_psm-grpc-server:47704-route-0"
}
]
}
]
},
"lastUpdated": "2023-09-12T20:09:28.178Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"versionInfo": "1694549355759946050",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230912-2005-h57dy"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230912-2005-h57dy",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-09-12T20:09:28.220Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"versionInfo": "3",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_3189496301170293366",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3806992279121263138_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.247",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.9.131",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.13.203",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-09-12T20:09:47.635Z",
"clientStatus": "ACKED"
}
]
}
I0912 20:09:50.339241 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0912 20:09:50.339374 139691887030272 xds_k8s_testcase.py:267] --- Starting subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0912 20:09:50.339526 139691887030272 grpc.py:74] [psm-grpc-client-54f9d6dc9-nttrs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:09:54.382772 139691887030272 xds_k8s_testcase.py:586] [psm-grpc-client-54f9d6dc9-nttrs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-68877867c4-wr5tg: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-68877867c4-wr5tg: 100

I0912 20:09:54.383131 139691887030272 xds_k8s_testcase.py:272] --- Finished subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0912 20:09:54.383306 139691887030272 xds_k8s_testcase.py:672] ----- TestMethod __main__.AffinityTest.test_affinity teardown -----
I0912 20:09:54.530618 139691887030272 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2005-h57dy"
I0912 20:09:55.425474 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549394534-6052f06b730b5-7e5e29f5-43c9a410
I0912 20:10:19.446059 139691887030272 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2005-h57dy"
I0912 20:10:19.578656 139691887030272 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:19.578884 139691887030272 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2005-h57dy"
I0912 20:10:19.972232 139691887030272 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:19.972515 139691887030272 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2005-h57dy"
I0912 20:10:20.522675 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549419976-6052f083b6595-3455a743-0a63c734
I0912 20:10:22.858036 139691887030272 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2005-h57dy"
I0912 20:10:23.260836 139691887030272 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:23.261057 139691887030272 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2005-h57dy"
I0912 20:10:23.956816 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549423266-6052f086d9a9f-28d2fbff-c6a3976c
I0912 20:10:26.313952 139691887030272 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2005-h57dy"
I0912 20:10:26.765253 139691887030272 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:26.765495 139691887030272 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2005-h57dy"
I0912 20:10:27.502350 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549426770-6052f08a313b3-d5f4b390-7b47dba6
I0912 20:10:53.561575 139691887030272 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2005-h57dy"
I0912 20:10:54.004781 139691887030272 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:54.005032 139691887030272 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2005-h57dy"
I0912 20:10:54.528423 139691887030272 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2005-h57dy" not deleted since it does not exist
I0912 20:10:54.528652 139691887030272 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2005-h57dy"
I0912 20:10:55.143969 139691887030272 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549454609-6052f0a4bdc76-6a27be55-c224bf41
I0912 20:10:59.693226 139691887030272 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 20:11:40.046061 139691887030272 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2005-h57dy/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:11:40.731004 139691887030272 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 20:11:40.858657 139691887030272 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2005-h57dy
I0912 20:12:11.042445 139691887030272 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 20:12:21.295793 139691887030272 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 20:12:21.413065 139691887030272 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2005-h57dy/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:12:22.129758 139691887030272 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 20:12:22.251086 139691887030272 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2005-h57dy
I0912 20:13:52.606969 139691887030272 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 20:13:52.607407 139691887030272 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-jg7abuwt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2005-h57dy%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-jg7abuwt%22;timeRange=2023-09-12T20:08:11.945512Z%2F2023-09-12T20:12:11.042393Z?project=grpc-testing
I0912 20:13:52.607650 139691887030272 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-5pop2vcc:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2005-h57dy%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-5pop2vcc%22;timeRange=2023-09-12T20:06:36.538205Z%2F2023-09-12T20:13:52.606797Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
I0912 20:13:52.607938 139691887030272 base_testcase.py:52] ----- TestCase __main__.AffinityTest.test_affinity PASSED -----
----------------------------------------------------------------------
Ran 1 test in 492.969s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/api_listener_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:13:53.955224 140575970586624 xds_k8s_testcase.py:155] ----- Testing ApiListenerTest -----
I0912 20:13:53.955390 140575970586624 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:13:53.955775 140575970586624 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:13:53.972866 140575970586624 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:13:53.989527 140575970586624 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0912 20:13:53.994443 140575970586624 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2013-gkfin
I0912 20:13:54.470328 140575970586624 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/api_listener_test
I0912 20:13:54.850826 140575970586624 xds_k8s_testcase.py:648] Found unused xds port: 13417
I0912 20:13:54.851141 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0912 20:13:54.851228 140575970586624 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2013-gkfin"
I0912 20:13:54.854395 140575970586624 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2013-gkfin
type: GRPC
...

I0912 20:13:55.584054 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549634916-6052f150b206b-7522be6e-d35577d6
I0912 20:13:58.104310 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0912 20:13:58.104488 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0912 20:13:58.104559 140575970586624 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2013-gkfin"
I0912 20:13:58.111845 140575970586624 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2013-gkfin
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2013-gkfin
protocol: GRPC
...

I0912 20:13:58.800334 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549638114-6052f153bec49-5445907f-aa084ce6
I0912 20:15:01.757629 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0912 20:15:01.757909 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0912 20:15:01.758014 140575970586624 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2013-gkfin": psm-grpc-server:13417 -> psm-interop-backend-service-20230912-2013-gkfin
I0912 20:15:01.769099 140575970586624 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2013-gkfin
hostRules:
- hosts:
- psm-grpc-server:13417
pathMatcher: psm-interop-path-matcher-20230912-2013-gkfin
name: psm-interop-url-map-20230912-2013-gkfin
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2013-gkfin
name: psm-interop-path-matcher-20230912-2013-gkfin
...

I0912 20:15:02.426908 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549701771-6052f19074046-a0692953-1e40465f
I0912 20:15:04.756184 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0912 20:15:04.756402 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0912 20:15:04.756510 140575970586624 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2013-gkfin proxy "GRPC" to URL map psm-interop-url-map-20230912-2013-gkfin
I0912 20:15:04.758589 140575970586624 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2013-gkfin
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2013-gkfin
validate_for_proxyless: true
...

I0912 20:15:05.407205 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549704760-6052f1934dd3a-66167430-aaf62aea
I0912 20:15:07.758770 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0912 20:15:07.759011 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0912 20:15:07.759117 140575970586624 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2013-gkfin" in network "default-vpc": 0.0.0.0:13417 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2013-gkfin
I0912 20:15:07.761756 140575970586624 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2013-gkfin
network: global/networks/default-vpc
portRange: 13417
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2013-gkfin
...

I0912 20:15:08.617308 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549707763-6052f1962b0e7-d05eeb0c-cc862bad
I0912 20:15:21.662624 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0912 20:15:21.662895 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0912 20:15:21.663003 140575970586624 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2013-gkfin: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:15:21.663270 140575970586624 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2013-gkfin%22;timeRange=2023-09-12T20:15:21.663101Z%2F2023-09-12T20:45:21.663101Z?project=grpc-testing
I0912 20:15:21.668304 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2013-gkfin
labels:
name: psm-interop-server-20230912-2013-gkfin
owner: xds-k8s-interop-test
...

I0912 20:15:21.805854 140575970586624 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2013-gkfin created
I0912 20:15:21.808887 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2013-gkfin
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2013-gkfin-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:15:21.878913 140575970586624 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:15:21.879150 140575970586624 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:15:21.975389 140575970586624 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2013-gkfin-psm-grpc-server in zones=['us-central1-a']
I0912 20:15:21.975627 140575970586624 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2013-gkfin/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:15:22.699714 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2013-gkfin
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:15:22.758772 140575970586624 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:15:22.766919 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2013-gkfin
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-fnlzmc9n
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-fnlzmc9n
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-fnlzmc9n
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2013-gkfin-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:15:22.841442 140575970586624 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:15:22.841688 140575970586624 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0912 20:15:22.953604 140575970586624 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-7d8cc95978-v6pdf']
I0912 20:15:22.953876 140575970586624 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-7d8cc95978-v6pdf to start
I0912 20:15:26.191937 140575970586624 k8s_base_runner.py:628] Pod psm-grpc-server-7d8cc95978-v6pdf ready, IP: 10.120.10.211
I0912 20:15:26.192218 140575970586624 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-7d8cc95978-v6pdf to api_listener_test/test_app_logs/psm-interop-server-20230912-2013-gkfin_psm-grpc-server-7d8cc95978-v6pdf.log
I0912 20:15:26.192697 140575880242752 k8s_log_collector.py:70] Starting log collection thread 140575880242752 for psm-grpc-server-7d8cc95978-v6pdf
I0912 20:15:26.192862 140575970586624 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0912 20:15:32.655114 140575970586624 k8s_base_runner.py:596] Deployment psm-grpc-server has 1 replicas available
I0912 20:15:32.655543 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0912 20:15:32.655654 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0912 20:15:32.712912 140575970586624 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:15:32.846518 140575970586624 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2013-gkfin-psm-grpc-server" in zone us-central1-a
I0912 20:15:32.846773 140575970586624 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2013-gkfin: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2013-gkfin-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2013-gkfin-psm-grpc-server', zone='us-central1-a')}
I0912 20:15:32.850837 140575970586624 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2013-gkfin-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 20:15:33.935083 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549732853-6052f1ae18666-0f6709a4-3620743e
I0912 20:16:28.504532 140575970586624 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2013-gkfin to report all backends healthy: ['psm-interop-server-20230912-2013-gkfin-psm-grpc-server']
I0912 20:16:28.735927 140575970586624 compute.py:487] Backend psm-interop-server-20230912-2013-gkfin-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:16:32.740983 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0912 20:16:32.741217 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0912 20:16:32.741326 140575970586624 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2013-gkfin: server_target=xds:///psm-grpc-server:13417 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 20:16:32.741548 140575970586624 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2013-gkfin%22;timeRange=2023-09-12T20:16:32.741408Z%2F2023-09-12T20:46:32.741408Z?project=grpc-testing
I0912 20:16:32.743661 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2013-gkfin
labels:
name: psm-interop-client-20230912-2013-gkfin
owner: xds-k8s-interop-test
...

I0912 20:16:32.811301 140575970586624 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2013-gkfin created
I0912 20:16:32.811528 140575970586624 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2013-gkfin/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:16:33.680550 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2013-gkfin
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:16:33.747322 140575970586624 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:16:33.755256 140575970586624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2013-gkfin
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ls6f633k
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ls6f633k
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ls6f633k
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13417"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:16:33.834808 140575970586624 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:16:33.835047 140575970586624 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:16:33.998210 140575970586624 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7458fbd8cd-zj2gq']
I0912 20:16:33.998524 140575970586624 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-7458fbd8cd-zj2gq to start
I0912 20:16:46.629984 140575970586624 k8s_base_runner.py:628] Pod psm-grpc-client-7458fbd8cd-zj2gq ready, IP: 10.120.9.134
I0912 20:16:46.630291 140575970586624 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-7458fbd8cd-zj2gq to api_listener_test/test_app_logs/psm-interop-client-20230912-2013-gkfin_psm-grpc-client-7458fbd8cd-zj2gq.log
I0912 20:16:46.630707 140575871850048 k8s_log_collector.py:70] Starting log collection thread 140575871850048 for psm-grpc-client-7458fbd8cd-zj2gq
I0912 20:16:46.630858 140575970586624 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:16:49.857386 140575970586624 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:16:49.857964 140575970586624 client_app.py:200] [psm-grpc-client-7458fbd8cd-zj2gq] Waiting to report a READY channel to xds:///psm-grpc-server:13417
I0912 20:16:49.975269 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:00.058417 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:10.139198 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:20.219326 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:30.300434 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:46.391900 140575970586624 client_app.py:236] [psm-grpc-client-7458fbd8cd-zj2gq] Server channel:
I0912 20:17:46.428683 140575970586624 client_app.py:249] [psm-grpc-client-7458fbd8cd-zj2gq] Found subchannel in state READY:
I0912 20:17:46.429020 140575970586624 client_app.py:211] [psm-grpc-client-7458fbd8cd-zj2gq] Channel to xds:///psm-grpc-server:13417 transitioned to state READY:
I0912 20:17:46.429156 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0912 20:17:46.429342 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0912 20:17:46.429583 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:17:46.471954 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0912 20:17:46.472264 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0912 20:17:46.472528 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:17:50.531646 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:17:50.531955 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0912 20:17:50.532054 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0912 20:17:50.532145 140575970586624 traffic_director.py:536] Creating alternative URL map "psm-interop-url-map-alt-20230912-2013-gkfin": psm-grpc-server:13417 -> psm-interop-backend-service-20230912-2013-gkfin
I0912 20:17:50.537556 140575970586624 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2013-gkfin
hostRules:
- hosts:
- psm-grpc-server:13417
pathMatcher: psm-interop-path-matcher-20230912-2013-gkfin
name: psm-interop-url-map-alt-20230912-2013-gkfin
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2013-gkfin
name: psm-interop-path-matcher-20230912-2013-gkfin
...

I0912 20:17:51.202919 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549870543-6052f2316819f-52784e72-de645f7a
I0912 20:17:53.572233 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0912 20:17:53.572465 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0912 20:17:53.572557 140575970586624 traffic_director.py:609] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20230912-2013-gkfin" to URL map psm-interop-url-map-alt-20230912-2013-gkfin
I0912 20:17:53.574747 140575970586624 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20230912-2013-gkfin
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20230912-2013-gkfin
validate_for_proxyless: false
...

I0912 20:17:54.086505 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549873577-6052f2344cc34-1a11ea86-d1a73309
I0912 20:17:56.414710 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0912 20:17:56.414940 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0912 20:17:56.415042 140575970586624 traffic_director.py:679] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20230912-2013-gkfin" in network "default-vpc": 10.10.10.10:13417 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20230912-2013-gkfin
I0912 20:17:56.417638 140575970586624 compute.py:530] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20230912-2013-gkfin
network: global/networks/default-vpc
portRange: 13417
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20230912-2013-gkfin
...

I0912 20:17:57.281786 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549876423-6052f23703967-654ce079-862909fd
I0912 20:18:08.275433 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0912 20:18:08.275815 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0912 20:18:08.276013 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:18:12.367640 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:18:12.368088 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:18:12.402473 140575970586624 api_listener_test.py:110] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/6593166c-4589-4305-a330-c33749fcb2da",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.9.134",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.8.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:13417",
"versionInfo": "1694549836341869248",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:13417",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230912-2013-gkfin_0_psm-grpc-server:13417"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-09-12T20:17:30.366Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2013-gkfin_0_psm-grpc-server:13417",
"versionInfo": "1694549836341869248",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2013-gkfin_0_psm-grpc-server:13417",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:13417"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_5621515442266667113",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230912-2013-gkfin_0_psm-grpc-server:13417-route-0"
}
]
}
]
},
"lastUpdated": "2023-09-12T20:17:30.391Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5621515442266667113",
"versionInfo": "1694549836341869248",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5621515442266667113",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230912-2013-gkfin"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230912-2013-gkfin",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-09-12T20:17:30.415Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_5621515442266667113",
"versionInfo": "1",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_5621515442266667113",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_5475062765484603445_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.10.211",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-09-12T20:17:31.034Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1694549836341869248
I0912 20:18:12.403262 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0912 20:18:12.403393 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0912 20:18:12.403553 140575970586624 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2013-gkfin"
I0912 20:18:13.135963 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549892407-6052f24642139-4ae190e8-504dd1c9
I0912 20:18:37.092062 140575970586624 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2013-gkfin"
I0912 20:18:37.783847 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549917095-6052f25dcd5e9-b59d6917-d2458ce6
I0912 20:18:40.134752 140575970586624 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2013-gkfin"
I0912 20:18:40.898229 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549920140-6052f260b4e57-533b6224-e8017a28
I0912 20:18:43.239304 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0912 20:18:43.239545 140575970586624 xds_k8s_testcase.py:267] --- Starting subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0912 20:18:43.239968 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:18:47.293175 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:18:47.293553 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:18:47.327797 140575970586624 xds_k8s_testcase.py:543] Routing config not propagated yet. Retrying.
I0912 20:18:47.328352 140575970586624 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0912 20:18:49.330765 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:18:53.399212 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:18:53.399498 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:18:53.434054 140575970586624 xds_k8s_testcase.py:543] Routing config not propagated yet. Retrying.
I0912 20:18:53.434401 140575970586624 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0912 20:18:55.436848 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:18:59.502795 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:18:59.503138 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:18:59.537219 140575970586624 xds_k8s_testcase.py:543] Routing config not propagated yet. Retrying.
I0912 20:18:59.537684 140575970586624 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0912 20:19:01.540096 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:19:05.639241 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:19:05.639678 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:19:05.673871 140575970586624 xds_k8s_testcase.py:543] Routing config not propagated yet. Retrying.
I0912 20:19:05.674358 140575970586624 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0912 20:19:07.676758 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:19:11.778158 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:19:11.778552 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:19:11.812949 140575970586624 grpc.py:74] [psm-grpc-client-7458fbd8cd-zj2gq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:19:15.890306 140575970586624 xds_k8s_testcase.py:586] [psm-grpc-client-7458fbd8cd-zj2gq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d8cc95978-v6pdf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7d8cc95978-v6pdf: 100

I0912 20:19:15.890708 140575970586624 xds_k8s_testcase.py:552] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1694549936293589767
I0912 20:19:15.890940 140575970586624 xds_k8s_testcase.py:272] --- Finished subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0912 20:19:15.891096 140575970586624 xds_k8s_testcase.py:672] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0912 20:19:16.109409 140575970586624 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2013-gkfin"
I0912 20:19:16.254519 140575970586624 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:19:16.254755 140575970586624 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2013-gkfin"
I0912 20:19:17.068222 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549956258-6052f28326922-95f37b88-247f7016
I0912 20:19:30.130557 140575970586624 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2013-gkfin"
I0912 20:19:30.615833 140575970586624 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:19:30.616068 140575970586624 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2013-gkfin"
I0912 20:19:30.946220 140575970586624 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:19:30.946448 140575970586624 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2013-gkfin"
I0912 20:19:31.640240 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549970949-6052f291295eb-3acdf7ef-0b132b11
I0912 20:19:34.007488 140575970586624 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2013-gkfin"
I0912 20:19:34.495444 140575970586624 compute.py:561] Resource urlMap "psm-interop-url-map-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:19:34.495656 140575970586624 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2013-gkfin"
I0912 20:19:35.159218 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549974502-6052f2948ca96-7dbd5ce6-f9755dfb
I0912 20:19:37.477781 140575970586624 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2013-gkfin"
I0912 20:19:38.117829 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694549977482-6052f297646b3-fc8a88fa-ce6ff439
I0912 20:20:04.308572 140575970586624 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2013-gkfin"
I0912 20:20:04.749136 140575970586624 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:20:04.749384 140575970586624 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2013-gkfin"
I0912 20:20:05.170788 140575970586624 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2013-gkfin" not deleted since it does not exist
I0912 20:20:05.171007 140575970586624 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2013-gkfin"
I0912 20:20:05.698011 140575970586624 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550005255-6052f2b1e0ee5-9e12f1ad-23815565
I0912 20:20:10.229848 140575970586624 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 20:20:50.639392 140575970586624 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2013-gkfin/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:20:51.321533 140575970586624 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 20:20:51.445660 140575970586624 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2013-gkfin
I0912 20:21:21.683857 140575970586624 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 20:21:31.928238 140575970586624 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 20:21:32.059653 140575970586624 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2013-gkfin/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:21:32.582190 140575970586624 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 20:21:32.698903 140575970586624 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2013-gkfin
I0912 20:22:32.956994 140575970586624 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 20:22:32.957407 140575970586624 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-ls6f633k:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2013-gkfin%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-ls6f633k%22;timeRange=2023-09-12T20:16:32.741400Z%2F2023-09-12T20:21:21.683807Z?project=grpc-testing
I0912 20:22:32.957571 140575970586624 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-fnlzmc9n:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2013-gkfin%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-fnlzmc9n%22;timeRange=2023-09-12T20:15:21.663092Z%2F2023-09-12T20:22:32.956820Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
I0912 20:22:32.957881 140575970586624 base_testcase.py:52] ----- TestCase __main__.ApiListenerTest.test_api_listener PASSED -----
----------------------------------------------------------------------
Ran 1 test in 519.006s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/baseline_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:22:34.337403 140646746947584 xds_k8s_testcase.py:155] ----- Testing BaselineTest -----
I0912 20:22:34.337551 140646746947584 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:22:34.337912 140646746947584 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:22:34.354259 140646746947584 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:22:34.370645 140646746947584 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] BaselineTest.test_traffic_director_grpc_setup
I0912 20:22:34.375379 140646746947584 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2022-qxl38
I0912 20:22:34.812074 140646746947584 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/baseline_test
I0912 20:22:35.213601 140646746947584 xds_k8s_testcase.py:648] Found unused xds port: 63005
I0912 20:22:35.213952 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0912 20:22:35.214039 140646746947584 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2022-qxl38"
I0912 20:22:35.217138 140646746947584 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2022-qxl38
type: GRPC
...

I0912 20:22:35.941725 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550155287-6052f340f5bcd-de596574-eba58c74
I0912 20:22:38.333866 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0912 20:22:38.334131 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0912 20:22:38.334220 140646746947584 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2022-qxl38"
I0912 20:22:38.341299 140646746947584 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2022-qxl38
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2022-qxl38
protocol: GRPC
...

I0912 20:22:39.026681 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550158343-6052f343dfd5a-b5e15e1c-ee07fa45
I0912 20:22:54.371428 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0912 20:22:54.371719 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0912 20:22:54.371837 140646746947584 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2022-qxl38": psm-grpc-server:63005 -> psm-interop-backend-service-20230912-2022-qxl38
I0912 20:22:54.382997 140646746947584 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2022-qxl38
hostRules:
- hosts:
- psm-grpc-server:63005
pathMatcher: psm-interop-path-matcher-20230912-2022-qxl38
name: psm-interop-url-map-20230912-2022-qxl38
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2022-qxl38
name: psm-interop-path-matcher-20230912-2022-qxl38
...

I0912 20:22:54.980685 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550174385-6052f3532c4ff-cf937343-4f72fbe8
I0912 20:22:57.281003 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0912 20:22:57.281230 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0912 20:22:57.281319 140646746947584 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2022-qxl38 proxy "GRPC" to URL map psm-interop-url-map-20230912-2022-qxl38
I0912 20:22:57.283263 140646746947584 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2022-qxl38
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2022-qxl38
validate_for_proxyless: true
...

I0912 20:22:57.890998 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550177285-6052f355f0477-3c903817-d5add43a
I0912 20:23:00.206676 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0912 20:23:00.206927 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0912 20:23:00.207027 140646746947584 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2022-qxl38" in network "default-vpc": 0.0.0.0:63005 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2022-qxl38
I0912 20:23:00.209573 140646746947584 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2022-qxl38
network: global/networks/default-vpc
portRange: 63005
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2022-qxl38
...

I0912 20:23:00.992193 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550180212-6052f358badbf-0b14b077-dcb33c7f
I0912 20:23:44.359453 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0912 20:23:44.359704 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0912 20:23:44.359850 140646746947584 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2022-qxl38: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:23:44.360099 140646746947584 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2022-qxl38%22;timeRange=2023-09-12T20:23:44.359932Z%2F2023-09-12T20:53:44.359932Z?project=grpc-testing
I0912 20:23:44.364956 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2022-qxl38
labels:
name: psm-interop-server-20230912-2022-qxl38
owner: xds-k8s-interop-test
...

I0912 20:23:44.519279 140646746947584 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2022-qxl38 created
I0912 20:23:44.522228 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2022-qxl38
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2022-qxl38-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:23:44.592862 140646746947584 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:23:44.593104 140646746947584 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:23:44.685810 140646746947584 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2022-qxl38-psm-grpc-server in zones=['us-central1-a']
I0912 20:23:44.686018 140646746947584 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2022-qxl38/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:23:45.416768 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2022-qxl38
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:23:45.491804 140646746947584 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:23:45.499824 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2022-qxl38
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zy95ndwx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zy95ndwx
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zy95ndwx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2022-qxl38-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:23:45.568082 140646746947584 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:23:45.568325 140646746947584 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0912 20:23:45.674969 140646746947584 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-788b4848b4-nt65z']
I0912 20:23:45.675230 140646746947584 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-788b4848b4-nt65z to start
I0912 20:23:48.908493 140646746947584 k8s_base_runner.py:628] Pod psm-grpc-server-788b4848b4-nt65z ready, IP: 10.120.9.135
I0912 20:23:48.908763 140646746947584 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-788b4848b4-nt65z to baseline_test/test_app_logs/psm-interop-server-20230912-2022-qxl38_psm-grpc-server-788b4848b4-nt65z.log
I0912 20:23:48.909205 140646657025600 k8s_log_collector.py:70] Starting log collection thread 140646657025600 for psm-grpc-server-788b4848b4-nt65z
I0912 20:23:48.909348 140646746947584 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0912 20:23:55.324489 140646746947584 k8s_base_runner.py:596] Deployment psm-grpc-server has 1 replicas available
I0912 20:23:55.324905 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0912 20:23:55.325030 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0912 20:23:55.361722 140646746947584 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:23:55.648833 140646746947584 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2022-qxl38-psm-grpc-server" in zone us-central1-a
I0912 20:23:55.649085 140646746947584 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2022-qxl38: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2022-qxl38-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2022-qxl38-psm-grpc-server', zone='us-central1-a')}
I0912 20:23:55.653169 140646746947584 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2022-qxl38-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 20:23:56.630862 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550235655-6052f38d9ad9f-5a5dffc1-655326ef
I0912 20:24:55.385827 140646746947584 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2022-qxl38 to report all backends healthy: ['psm-interop-server-20230912-2022-qxl38-psm-grpc-server']
I0912 20:25:08.339467 140646746947584 compute.py:487] Backend psm-interop-server-20230912-2022-qxl38-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:25:12.344212 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0912 20:25:12.344457 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0912 20:25:12.344559 140646746947584 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2022-qxl38: server_target=xds:///psm-grpc-server:63005 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 20:25:12.344791 140646746947584 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2022-qxl38%22;timeRange=2023-09-12T20:25:12.344635Z%2F2023-09-12T20:55:12.344635Z?project=grpc-testing
I0912 20:25:12.346818 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2022-qxl38
labels:
name: psm-interop-client-20230912-2022-qxl38
owner: xds-k8s-interop-test
...

I0912 20:25:12.388726 140646746947584 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2022-qxl38 created
I0912 20:25:12.388959 140646746947584 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2022-qxl38/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:25:13.030273 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2022-qxl38
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:25:13.105271 140646746947584 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:25:13.113064 140646746947584 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2022-qxl38
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-iewe01ex
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-iewe01ex
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-iewe01ex
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:63005"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:25:13.186782 140646746947584 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:25:13.187029 140646746947584 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:25:13.357300 140646746947584 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-85fd84996-2878h']
I0912 20:25:13.357582 140646746947584 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-85fd84996-2878h to start
I0912 20:25:25.993124 140646746947584 k8s_base_runner.py:628] Pod psm-grpc-client-85fd84996-2878h ready, IP: 10.120.14.240
I0912 20:25:25.993390 140646746947584 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-85fd84996-2878h to baseline_test/test_app_logs/psm-interop-client-20230912-2022-qxl38_psm-grpc-client-85fd84996-2878h.log
I0912 20:25:25.993821 140646648632896 k8s_log_collector.py:70] Starting log collection thread 140646648632896 for psm-grpc-client-85fd84996-2878h
I0912 20:25:25.994022 140646746947584 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:25:29.285172 140646746947584 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:25:29.285745 140646746947584 client_app.py:200] [psm-grpc-client-85fd84996-2878h] Waiting to report a READY channel to xds:///psm-grpc-server:63005
I0912 20:25:29.405074 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:25:39.489634 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:25:49.571279 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:25:59.654154 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:26:09.736156 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:26:25.824485 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:26:50.922023 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:27:16.018945 140646746947584 client_app.py:236] [psm-grpc-client-85fd84996-2878h] Server channel:
I0912 20:27:16.057118 140646746947584 client_app.py:249] [psm-grpc-client-85fd84996-2878h] Found subchannel in state READY:
I0912 20:27:16.057420 140646746947584 client_app.py:211] [psm-grpc-client-85fd84996-2878h] Channel to xds:///psm-grpc-server:63005 transitioned to state READY:
I0912 20:27:16.057552 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0912 20:27:16.057786 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.8_test_client_xds_config_exists ---
I0912 20:27:16.058036 140646746947584 grpc.py:74] [psm-grpc-client-85fd84996-2878h:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:27:16.100229 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.8_test_client_xds_config_exists ---
I0912 20:27:16.100553 140646746947584 xds_k8s_testcase.py:267] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.9_test_server_received_rpcs_from_test_client ---
I0912 20:27:16.100859 140646746947584 grpc.py:74] [psm-grpc-client-85fd84996-2878h:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 20:27:20.164267 140646746947584 xds_k8s_testcase.py:586] [psm-grpc-client-85fd84996-2878h] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-788b4848b4-nt65z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-788b4848b4-nt65z: 100

I0912 20:27:20.164647 140646746947584 xds_k8s_testcase.py:272] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.9_test_server_received_rpcs_from_test_client ---
I0912 20:27:20.164783 140646746947584 xds_k8s_testcase.py:672] ----- TestMethod __main__.BaselineTest.test_traffic_director_grpc_setup teardown -----
I0912 20:27:20.374980 140646746947584 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2022-qxl38"
I0912 20:27:21.157755 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550440379-6052f450d8347-57ff7904-da5bf7f9
I0912 20:27:47.448199 140646746947584 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2022-qxl38"
I0912 20:27:47.590357 140646746947584 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:27:47.590575 140646746947584 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2022-qxl38"
I0912 20:27:47.993309 140646746947584 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:27:47.993543 140646746947584 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2022-qxl38"
I0912 20:27:48.623403 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550467996-6052f46b2ece1-fa061c31-6e2942c6
I0912 20:27:50.968702 140646746947584 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2022-qxl38"
I0912 20:27:51.263878 140646746947584 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:27:51.264102 140646746947584 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2022-qxl38"
I0912 20:27:51.955038 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550471269-6052f46e4dd82-9a55dffb-8aeb8d01
I0912 20:27:54.311815 140646746947584 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2022-qxl38"
I0912 20:27:54.881161 140646746947584 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:27:54.881382 140646746947584 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2022-qxl38"
I0912 20:27:55.609394 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550474886-6052f471c0e9c-b13b12d6-db4492fa
I0912 20:28:23.924644 140646746947584 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2022-qxl38"
I0912 20:28:24.405134 140646746947584 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:28:24.405367 140646746947584 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2022-qxl38"
I0912 20:28:24.904886 140646746947584 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2022-qxl38" not deleted since it does not exist
I0912 20:28:24.905124 140646746947584 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2022-qxl38"
I0912 20:28:25.521743 140646746947584 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550505014-6052f48e7c507-a015d4d4-a9a9886a
I0912 20:28:30.027783 140646746947584 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 20:29:10.435848 140646746947584 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2022-qxl38/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:29:11.029407 140646746947584 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 20:29:11.157455 140646746947584 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2022-qxl38
I0912 20:29:41.386705 140646746947584 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 20:29:51.562262 140646746947584 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 20:29:51.682216 140646746947584 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2022-qxl38/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:29:52.416519 140646746947584 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 20:29:52.533182 140646746947584 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2022-qxl38
I0912 20:31:22.853649 140646746947584 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 20:31:22.854056 140646746947584 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-iewe01ex:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2022-qxl38%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-iewe01ex%22;timeRange=2023-09-12T20:25:12.344629Z%2F2023-09-12T20:29:41.386653Z?project=grpc-testing
I0912 20:31:22.854247 140646746947584 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-zy95ndwx:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2022-qxl38%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-zy95ndwx%22;timeRange=2023-09-12T20:23:44.359925Z%2F2023-09-12T20:31:22.853465Z?project=grpc-testing
[ OK ] BaselineTest.test_traffic_director_grpc_setup
I0912 20:31:22.854534 140646746947584 base_testcase.py:52] ----- TestCase __main__.BaselineTest.test_traffic_director_grpc_setup PASSED -----
----------------------------------------------------------------------
Ran 1 test in 528.520s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/change_backend_service_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:31:24.209692 140207297662976 xds_k8s_testcase.py:155] ----- Testing ChangeBackendServiceTest -----
I0912 20:31:24.209857 140207297662976 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:31:24.210445 140207297662976 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:31:24.227676 140207297662976 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:31:24.244503 140207297662976 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0912 20:31:24.249502 140207297662976 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2031-40kks
I0912 20:31:24.591607 140207297662976 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/change_backend_service_test
I0912 20:31:24.995950 140207297662976 xds_k8s_testcase.py:648] Found unused xds port: 56612
I0912 20:31:24.998572 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0912 20:31:24.998780 140207297662976 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2031-40kks"
I0912 20:31:25.001872 140207297662976 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2031-40kks
type: GRPC
...

I0912 20:31:25.702350 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550685071-6052f53a336d7-f8f6b495-86cefdf9
I0912 20:31:28.083627 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0912 20:31:28.083883 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0912 20:31:28.083992 140207297662976 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2031-40kks"
I0912 20:31:28.091125 140207297662976 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2031-40kks
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2031-40kks
protocol: GRPC
...

I0912 20:31:28.904359 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550688147-6052f53d22944-123cc3f0-46623fe9
I0912 20:32:29.870893 140207297662976 traffic_director.py:305] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20230912-2031-40kks"
I0912 20:32:29.875571 140207297662976 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2031-40kks
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20230912-2031-40kks
protocol: GRPC
...

I0912 20:32:30.585429 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550749877-6052f5780160a-b61cf2b9-bbdac6de
I0912 20:33:11.964851 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0912 20:33:11.965109 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0912 20:33:11.965217 140207297662976 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2031-40kks": psm-grpc-server:56612 -> psm-interop-backend-service-20230912-2031-40kks
I0912 20:33:11.976820 140207297662976 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2031-40kks
hostRules:
- hosts:
- psm-grpc-server:56612
pathMatcher: psm-interop-path-matcher-20230912-2031-40kks
name: psm-interop-url-map-20230912-2031-40kks
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2031-40kks
name: psm-interop-path-matcher-20230912-2031-40kks
...

I0912 20:33:12.626897 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550791979-6052f5a02801a-bedb0adb-28a61b05
I0912 20:33:14.960896 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0912 20:33:14.961119 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0912 20:33:14.961210 140207297662976 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2031-40kks proxy "GRPC" to URL map psm-interop-url-map-20230912-2031-40kks
I0912 20:33:14.963331 140207297662976 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2031-40kks
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2031-40kks
validate_for_proxyless: true
...

I0912 20:33:15.584749 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550794965-6052f5a3011f9-fb4d0823-51c47a26
I0912 20:33:17.964686 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0912 20:33:17.964923 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0912 20:33:17.965027 140207297662976 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2031-40kks" in network "default-vpc": 0.0.0.0:56612 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2031-40kks
I0912 20:33:17.967621 140207297662976 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2031-40kks
network: global/networks/default-vpc
portRange: 56612
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2031-40kks
...

I0912 20:33:18.776779 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550797969-6052f5a5de98f-b3b856bb-6231c559
I0912 20:34:12.543076 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0912 20:34:12.543363 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0912 20:34:12.543476 140207297662976 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2031-40kks: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:34:12.543731 140207297662976 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2031-40kks%22;timeRange=2023-09-12T20:34:12.543557Z%2F2023-09-12T21:04:12.543557Z?project=grpc-testing
I0912 20:34:12.549488 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2031-40kks
labels:
name: psm-interop-server-20230912-2031-40kks
owner: xds-k8s-interop-test
...

I0912 20:34:12.699497 140207297662976 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2031-40kks created
I0912 20:34:12.702400 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2031-40kks
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2031-40kks-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:34:12.770250 140207297662976 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:34:12.770466 140207297662976 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:34:12.869996 140207297662976 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2031-40kks-psm-grpc-server in zones=['us-central1-a']
I0912 20:34:12.870248 140207297662976 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2031-40kks/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:34:13.658539 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2031-40kks
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:34:13.724932 140207297662976 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:34:13.733008 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2031-40kks
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2wjbpjfm
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2wjbpjfm
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2wjbpjfm
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2031-40kks-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:34:13.803247 140207297662976 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:34:13.803529 140207297662976 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0912 20:34:13.911721 140207297662976 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-74df897499-bp29z']
I0912 20:34:13.911993 140207297662976 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-74df897499-bp29z to start
I0912 20:34:17.152965 140207297662976 k8s_base_runner.py:628] Pod psm-grpc-server-74df897499-bp29z ready, IP: 10.120.16.141
I0912 20:34:17.153239 140207297662976 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-74df897499-bp29z to change_backend_service_test/test_app_logs/psm-interop-server-20230912-2031-40kks_psm-grpc-server-74df897499-bp29z.log
I0912 20:34:17.153712 140207206737472 k8s_log_collector.py:70] Starting log collection thread 140207206737472 for psm-grpc-server-74df897499-bp29z
I0912 20:34:17.153880 140207297662976 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0912 20:34:24.615278 140207297662976 k8s_base_runner.py:596] Deployment psm-grpc-server has 1 replicas available
I0912 20:34:24.615680 140207297662976 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230912-2031-40kks: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:34:24.615911 140207297662976 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2031-40kks%22;timeRange=2023-09-12T20:34:24.615794Z%2F2023-09-12T21:04:24.615794Z?project=grpc-testing
I0912 20:34:24.615989 140207297662976 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20230912-2031-40kks
I0912 20:34:24.674301 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2031-40kks
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:34:24.719605 140207297662976 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0912 20:34:24.719826 140207297662976 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server-alt
I0912 20:34:24.820611 140207297662976 k8s_base_runner.py:675] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt in zones=['us-central1-a']
I0912 20:34:24.820842 140207297662976 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2031-40kks/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:34:25.444611 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2031-40kks
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:34:25.484206 140207297662976 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0912 20:34:25.492182 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2031-40kks
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-jh30lhtj
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-jh30lhtj
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-jh30lhtj
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:34:25.543907 140207297662976 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0912 20:34:25.544178 140207297662976 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0912 20:34:25.630604 140207297662976 k8s_base_runner.py:615] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:34:25.630870 140207297662976 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-alt-5fd594d4d4-njkr7 to start
I0912 20:34:27.894145 140207297662976 k8s_base_runner.py:628] Pod psm-grpc-server-alt-5fd594d4d4-njkr7 ready, IP: 10.120.13.209
I0912 20:34:27.894413 140207297662976 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-alt-5fd594d4d4-njkr7 to change_backend_service_test/test_app_logs/psm-interop-server-20230912-2031-40kks_psm-grpc-server-alt-5fd594d4d4-njkr7.log
I0912 20:34:27.894842 140207127000640 k8s_log_collector.py:70] Starting log collection thread 140207127000640 for psm-grpc-server-alt-5fd594d4d4-njkr7
I0912 20:34:27.894984 140207297662976 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0912 20:34:35.348992 140207297662976 k8s_base_runner.py:596] Deployment psm-grpc-server-alt has 1 replicas available
I0912 20:34:35.349426 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0912 20:34:35.349540 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0912 20:34:35.386082 140207297662976 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:34:35.520703 140207297662976 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2031-40kks-psm-grpc-server" in zone us-central1-a
I0912 20:34:35.520946 140207297662976 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2031-40kks: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2031-40kks-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2031-40kks-psm-grpc-server', zone='us-central1-a')}
I0912 20:34:35.524982 140207297662976 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2031-40kks-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 20:34:36.298917 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550875527-6052f5efd5854-00447dec-66efdd11
I0912 20:35:34.976443 140207297662976 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2031-40kks to report all backends healthy: ['psm-interop-server-20230912-2031-40kks-psm-grpc-server']
I0912 20:35:39.373789 140207297662976 compute.py:487] Backend psm-interop-server-20230912-2031-40kks-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:35:43.417017 140207297662976 traffic_director.py:333] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:35:43.537077 140207297662976 traffic_director.py:336] Loaded NEG "psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt" in zone us-central1-a
I0912 20:35:43.537297 140207297662976 traffic_director.py:343] Adding backends to Backend Service psm-interop-backend-service-alt-20230912-2031-40kks: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt', zone='us-central1-a')}
I0912 20:35:43.541474 140207297662976 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2031-40kks-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0912 20:35:44.450994 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694550943544-6052f630b336f-e2029b17-5811bf31
I0912 20:36:38.281392 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0912 20:36:38.281698 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0912 20:36:38.281829 140207297662976 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2031-40kks: server_target=xds:///psm-grpc-server:56612 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 20:36:38.282077 140207297662976 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2031-40kks%22;timeRange=2023-09-12T20:36:38.281913Z%2F2023-09-12T21:06:38.281913Z?project=grpc-testing
I0912 20:36:38.284041 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2031-40kks
labels:
name: psm-interop-client-20230912-2031-40kks
owner: xds-k8s-interop-test
...

I0912 20:36:38.358502 140207297662976 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2031-40kks created
I0912 20:36:38.358766 140207297662976 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2031-40kks/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:36:39.114032 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2031-40kks
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:36:39.178343 140207297662976 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:36:39.186423 140207297662976 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2031-40kks
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-e9jj6q1k
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-e9jj6q1k
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-e9jj6q1k
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:56612"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:36:39.259369 140207297662976 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:36:39.259597 140207297662976 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:36:39.367553 140207297662976 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-756f867745-4lxtf']
I0912 20:36:39.367855 140207297662976 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-756f867745-4lxtf to start
I0912 20:36:42.605845 140207297662976 k8s_base_runner.py:628] Pod psm-grpc-client-756f867745-4lxtf ready, IP: 10.120.10.215
I0912 20:36:42.606141 140207297662976 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-756f867745-4lxtf to change_backend_service_test/test_app_logs/psm-interop-client-20230912-2031-40kks_psm-grpc-client-756f867745-4lxtf.log
I0912 20:36:42.606584 140207118607936 k8s_log_collector.py:70] Starting log collection thread 140207118607936 for psm-grpc-client-756f867745-4lxtf
I0912 20:36:42.606855 140207297662976 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:36:45.823258 140207297662976 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:36:45.823847 140207297662976 client_app.py:200] [psm-grpc-client-756f867745-4lxtf] Waiting to report a READY channel to xds:///psm-grpc-server:56612
I0912 20:36:45.946675 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:36:56.034583 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:37:06.118080 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:37:16.201486 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:37:26.285164 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:37:42.374034 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:38:07.471769 140207297662976 client_app.py:236] [psm-grpc-client-756f867745-4lxtf] Server channel:
I0912 20:38:07.510021 140207297662976 client_app.py:249] [psm-grpc-client-756f867745-4lxtf] Found subchannel in state READY:
I0912 20:38:07.510316 140207297662976 client_app.py:211] [psm-grpc-client-756f867745-4lxtf] Channel to xds:///psm-grpc-server:56612 transitioned to state READY:
I0912 20:38:07.510425 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0912 20:38:07.510480 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0912 20:38:07.510688 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:38:07.555855 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0912 20:38:07.556210 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0912 20:38:07.556465 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:11.622026 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-74df897499-bp29z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-74df897499-bp29z: 100

I0912 20:38:11.622408 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0912 20:38:11.622514 140207297662976 xds_k8s_testcase.py:267] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0912 20:38:11.622600 140207297662976 traffic_director.py:495] Patching URL map "psm-interop-url-map-20230912-2031-40kks": psm-grpc-server:56612 -> psm-interop-backend-service-alt-20230912-2031-40kks
I0912 20:38:11.628035 140207297662976 compute.py:537] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20230912-2031-40kks
hostRules:
- hosts:
- psm-grpc-server:56612
pathMatcher: psm-interop-path-matcher-20230912-2031-40kks
name: psm-interop-url-map-20230912-2031-40kks
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20230912-2031-40kks
name: psm-interop-path-matcher-20230912-2031-40kks
...

I0912 20:38:12.404656 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551091631-6052f6bded452-b7fcf026-4644b956
I0912 20:38:23.425482 140207297662976 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:38:23.425847 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:27.496044 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-74df897499-bp29z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-74df897499-bp29z: 100

I0912 20:38:27.496706 140207297662976 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-5fd594d4d4-njkr7' not found in {'psm-grpc-server-74df897499-bp29z': 100} : Server psm-grpc-server-alt-5fd594d4d4-njkr7 did not receive RPCs.
I0912 20:38:28.498121 140207297662976 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:38:28.498480 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:32.550939 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-74df897499-bp29z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-74df897499-bp29z: 100

I0912 20:38:32.551296 140207297662976 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-5fd594d4d4-njkr7' not found in {'psm-grpc-server-74df897499-bp29z': 100} : Server psm-grpc-server-alt-5fd594d4d4-njkr7 did not receive RPCs.
I0912 20:38:33.552756 140207297662976 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:38:33.553181 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:37.611589 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-74df897499-bp29z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-74df897499-bp29z: 100

I0912 20:38:37.612080 140207297662976 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-5fd594d4d4-njkr7' not found in {'psm-grpc-server-74df897499-bp29z': 100} : Server psm-grpc-server-alt-5fd594d4d4-njkr7 did not receive RPCs.
I0912 20:38:38.613400 140207297662976 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:38:38.613764 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:45.109086 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-74df897499-bp29z: 96
psm-grpc-server-alt-5fd594d4d4-njkr7: 4
rpcs_by_method:
UnaryCall:
psm-grpc-server-74df897499-bp29z: 96
psm-grpc-server-alt-5fd594d4d4-njkr7: 4

I0912 20:38:45.109408 140207297662976 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-74df897499-bp29z' not found in ['psm-grpc-server-alt-5fd594d4d4-njkr7'] : Unexpected server psm-grpc-server-74df897499-bp29z received RPCs.
I0912 20:38:46.110882 140207297662976 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-5fd594d4d4-njkr7']
I0912 20:38:46.111212 140207297662976 grpc.py:74] [psm-grpc-client-756f867745-4lxtf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:38:50.148511 140207297662976 xds_k8s_testcase.py:586] [psm-grpc-client-756f867745-4lxtf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-5fd594d4d4-njkr7: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-5fd594d4d4-njkr7: 100

I0912 20:38:50.148906 140207297662976 xds_k8s_testcase.py:272] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0912 20:38:50.149079 140207297662976 xds_k8s_testcase.py:672] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0912 20:38:50.353857 140207297662976 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2031-40kks"
I0912 20:38:51.271604 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551130357-6052f6e2dc0da-1c6b6847-97d10615
I0912 20:39:19.680088 140207297662976 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2031-40kks"
I0912 20:39:19.812738 140207297662976 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2031-40kks" not deleted since it does not exist
I0912 20:39:19.812978 140207297662976 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2031-40kks"
I0912 20:39:20.268481 140207297662976 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2031-40kks" not deleted since it does not exist
I0912 20:39:20.268727 140207297662976 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2031-40kks"
I0912 20:39:20.711781 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551160272-6052f6ff63657-be20c549-7d3ea8ac
I0912 20:39:23.035035 140207297662976 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2031-40kks"
I0912 20:39:23.344039 140207297662976 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2031-40kks" not deleted since it does not exist
I0912 20:39:23.344273 140207297662976 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2031-40kks"
I0912 20:39:24.048965 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551163349-6052f70252aa4-6d43691f-b4478de3
I0912 20:39:26.491330 140207297662976 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2031-40kks"
I0912 20:39:26.961203 140207297662976 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2031-40kks" not deleted since it does not exist
I0912 20:39:26.961444 140207297662976 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2031-40kks"
I0912 20:39:27.618741 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551166966-6052f705c5b2a-629e6602-9f21b71f
I0912 20:40:00.268873 140207297662976 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2031-40kks"
I0912 20:40:00.996049 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551200275-6052f72589ab8-9435d13c-dd2bc6f5
I0912 20:40:29.280699 140207297662976 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2031-40kks"
I0912 20:40:29.702457 140207297662976 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2031-40kks" not deleted since it does not exist
I0912 20:40:29.702730 140207297662976 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2031-40kks"
I0912 20:40:30.334868 140207297662976 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551229795-6052f741b0db4-7c6a275e-fee325a7
I0912 20:40:34.759250 140207297662976 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 20:41:15.275531 140207297662976 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2031-40kks/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:41:15.994914 140207297662976 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 20:41:16.120050 140207297662976 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2031-40kks
I0912 20:41:46.297879 140207297662976 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 20:41:56.473973 140207297662976 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 20:41:56.597902 140207297662976 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2031-40kks/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:41:57.208383 140207297662976 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 20:41:57.325257 140207297662976 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2031-40kks
I0912 20:44:57.884703 140207297662976 k8s_base_runner.py:535] Deleting deployment psm-grpc-server-alt
I0912 20:44:57.952647 140207297662976 k8s_base_runner.py:540] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0912 20:44:57.952980 140207297662976 k8s_base_runner.py:548] Deleting service psm-grpc-server-alt
I0912 20:44:58.015743 140207297662976 k8s_base_runner.py:552] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0912 20:44:58.016039 140207297662976 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2031-40kks/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:44:58.567696 140207297662976 k8s_base_runner.py:561] Deleting service account psm-grpc-server-alt
I0912 20:44:58.635178 140207297662976 k8s_base_runner.py:565] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0912 20:44:58.635439 140207297662976 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2031-40kks
I0912 20:44:58.704473 140207297662976 k8s_base_runner.py:577] Namespace psm-interop-server-20230912-2031-40kks deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230912-2031-40kks" not found
I0912 20:44:58.704964 140207297662976 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 20:44:58.705233 140207297662976 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-e9jj6q1k:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2031-40kks%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-e9jj6q1k%22;timeRange=2023-09-12T20:36:38.281902Z%2F2023-09-12T20:41:46.297828Z?project=grpc-testing
I0912 20:44:58.705406 140207297662976 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-2wjbpjfm:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2031-40kks%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-2wjbpjfm%22;timeRange=2023-09-12T20:34:12.543547Z%2F2023-09-12T20:44:57.884649Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
I0912 20:44:58.705692 140207297662976 base_testcase.py:52] ----- TestCase __main__.ChangeBackendServiceTest.test_change_backend_service PASSED -----
----------------------------------------------------------------------
Ran 1 test in 814.499s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:45:00.126674 140449033547776 xds_k8s_testcase.py:155] ----- Testing CustomLbTest -----
I0912 20:45:00.127015 140449033547776 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:45:00.127324 140449033547776 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:45:00.144796 140449033547776 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:45:00.162678 140449033547776 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0912 20:45:00.167464 140449033547776 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2045-bmll2
I0912 20:45:00.592606 140449033547776 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
I0912 20:45:00.907108 140449033547776 xds_k8s_testcase.py:648] Found unused xds port: 6693
I0912 20:45:00.907512 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0912 20:45:00.907643 140449033547776 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2045-bmll2"
I0912 20:45:00.910963 140449033547776 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2045-bmll2
type: GRPC
...

I0912 20:45:01.758249 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551500981-6052f844503ee-17e71c1d-4fbcd825
I0912 20:45:04.193501 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0912 20:45:04.193827 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0912 20:45:04.193941 140449033547776 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2045-bmll2"
I0912 20:45:04.202697 140449033547776 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2045-bmll2
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20230912-2045-bmll2
protocol: GRPC
...

I0912 20:45:04.917845 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551504205-6052f8476359b-832fc032-a86da3e5
I0912 20:45:24.835072 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0912 20:45:24.835318 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0912 20:45:24.835411 140449033547776 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2045-bmll2": psm-grpc-server:6693 -> psm-interop-backend-service-20230912-2045-bmll2
I0912 20:45:24.846338 140449033547776 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2045-bmll2
hostRules:
- hosts:
- psm-grpc-server:6693
pathMatcher: psm-interop-path-matcher-20230912-2045-bmll2
name: psm-interop-url-map-20230912-2045-bmll2
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2045-bmll2
name: psm-interop-path-matcher-20230912-2045-bmll2
...

I0912 20:45:25.601965 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551524848-6052f85b132bb-4a255fcc-988b8325
I0912 20:45:27.926768 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0912 20:45:27.927010 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0912 20:45:27.927102 140449033547776 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2045-bmll2 proxy "GRPC" to URL map psm-interop-url-map-20230912-2045-bmll2
I0912 20:45:27.929028 140449033547776 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2045-bmll2
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2045-bmll2
validate_for_proxyless: true
...

I0912 20:45:28.598069 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551527930-6052f85e03c69-49e72901-22a42d09
I0912 20:45:30.970480 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0912 20:45:30.970746 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0912 20:45:30.970867 140449033547776 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2045-bmll2" in network "default-vpc": 0.0.0.0:6693 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2045-bmll2
I0912 20:45:30.973414 140449033547776 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2045-bmll2
network: global/networks/default-vpc
portRange: 6693
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2045-bmll2
...

I0912 20:45:31.786204 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551530975-6052f860eb166-e9c04370-717cd029
I0912 20:46:21.430376 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0912 20:46:21.430684 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0912 20:46:21.430813 140449033547776 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2045-bmll2: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:46:21.431068 140449033547776 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2045-bmll2%22;timeRange=2023-09-12T20:46:21.430900Z%2F2023-09-12T21:16:21.430900Z?project=grpc-testing
I0912 20:46:21.436154 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2045-bmll2
labels:
name: psm-interop-server-20230912-2045-bmll2
owner: xds-k8s-interop-test
...

I0912 20:46:21.601455 140449033547776 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2045-bmll2 created
I0912 20:46:21.604386 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2045-bmll2
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2045-bmll2-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:46:21.678006 140449033547776 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:46:21.678221 140449033547776 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:46:21.785428 140449033547776 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2045-bmll2-psm-grpc-server in zones=['us-central1-a']
I0912 20:46:21.785648 140449033547776 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2045-bmll2/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:46:22.485870 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2045-bmll2
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:46:22.554781 140449033547776 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:46:22.563017 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2045-bmll2
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kqfj0jck
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kqfj0jck
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kqfj0jck
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2045-bmll2-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:46:22.640956 140449033547776 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:46:22.641196 140449033547776 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0912 20:46:22.763459 140449033547776 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-bf45c4fc7-bx26d']
I0912 20:46:22.763741 140449033547776 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-bf45c4fc7-bx26d to start
I0912 20:46:26.032927 140449033547776 k8s_base_runner.py:628] Pod psm-grpc-server-bf45c4fc7-bx26d ready, IP: 10.120.9.139
I0912 20:46:26.033203 140449033547776 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-bf45c4fc7-bx26d to custom_lb_test/test_app_logs/psm-interop-server-20230912-2045-bmll2_psm-grpc-server-bf45c4fc7-bx26d.log
I0912 20:46:26.033674 140448942777920 k8s_log_collector.py:70] Starting log collection thread 140448942777920 for psm-grpc-server-bf45c4fc7-bx26d
I0912 20:46:26.033858 140449033547776 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0912 20:46:32.428019 140449033547776 k8s_base_runner.py:596] Deployment psm-grpc-server has 1 replicas available
I0912 20:46:32.428438 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0912 20:46:32.428547 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0912 20:46:32.472014 140449033547776 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:46:32.601683 140449033547776 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2045-bmll2-psm-grpc-server" in zone us-central1-a
I0912 20:46:32.601918 140449033547776 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2045-bmll2: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2045-bmll2-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2045-bmll2-psm-grpc-server', zone='us-central1-a')}
I0912 20:46:32.605981 140449033547776 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2045-bmll2-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 20:46:33.529832 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551592608-6052f89bb23b1-af142f82-8454fd12
I0912 20:47:34.170967 140449033547776 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2045-bmll2 to report all backends healthy: ['psm-interop-server-20230912-2045-bmll2-psm-grpc-server']
I0912 20:47:51.112489 140449033547776 compute.py:487] Backend psm-interop-server-20230912-2045-bmll2-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:47:55.117328 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0912 20:47:55.117693 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0912 20:47:55.117843 140449033547776 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2045-bmll2: server_target=xds:///psm-grpc-server:6693 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 20:47:55.118075 140449033547776 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2045-bmll2%22;timeRange=2023-09-12T20:47:55.117930Z%2F2023-09-12T21:17:55.117930Z?project=grpc-testing
I0912 20:47:55.120155 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2045-bmll2
labels:
name: psm-interop-client-20230912-2045-bmll2
owner: xds-k8s-interop-test
...

I0912 20:47:55.169219 140449033547776 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2045-bmll2 created
I0912 20:47:55.169458 140449033547776 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2045-bmll2/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:47:55.879606 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2045-bmll2
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:47:55.948585 140449033547776 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:47:55.956750 140449033547776 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2045-bmll2
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y3evaujm
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y3evaujm
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y3evaujm
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:6693"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:47:56.037819 140449033547776 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:47:56.038076 140449033547776 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:47:56.215635 140449033547776 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5f9c45c7db-cx2mk']
I0912 20:47:56.215936 140449033547776 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-5f9c45c7db-cx2mk to start
I0912 20:47:59.486960 140449033547776 k8s_base_runner.py:628] Pod psm-grpc-client-5f9c45c7db-cx2mk ready, IP: 10.120.10.219
I0912 20:47:59.487230 140449033547776 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-5f9c45c7db-cx2mk to custom_lb_test/test_app_logs/psm-interop-client-20230912-2045-bmll2_psm-grpc-client-5f9c45c7db-cx2mk.log
I0912 20:47:59.487702 140448934385216 k8s_log_collector.py:70] Starting log collection thread 140448934385216 for psm-grpc-client-5f9c45c7db-cx2mk
I0912 20:47:59.487856 140449033547776 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:48:02.743274 140449033547776 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:48:02.743882 140449033547776 client_app.py:200] [psm-grpc-client-5f9c45c7db-cx2mk] Waiting to report a READY channel to xds:///psm-grpc-server:6693
I0912 20:48:02.869743 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:48:12.956926 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:48:23.040023 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:48:33.124184 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:48:43.208344 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:48:59.298005 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:49:24.396823 140449033547776 client_app.py:236] [psm-grpc-client-5f9c45c7db-cx2mk] Server channel:
I0912 20:49:24.435539 140449033547776 client_app.py:249] [psm-grpc-client-5f9c45c7db-cx2mk] Found subchannel in state READY:
I0912 20:49:24.435857 140449033547776 client_app.py:211] [psm-grpc-client-5f9c45c7db-cx2mk] Channel to xds:///psm-grpc-server:6693 transitioned to state READY:
I0912 20:49:24.435998 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0912 20:49:24.436078 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0912 20:49:24.436293 140449033547776 grpc.py:74] [psm-grpc-client-5f9c45c7db-cx2mk:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 20:49:24.483916 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0912 20:49:24.484276 140449033547776 xds_k8s_testcase.py:267] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0912 20:49:24.484657 140449033547776 grpc.py:74] [psm-grpc-client-5f9c45c7db-cx2mk:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0912 20:49:34.535772 140449033547776 grpc.py:74] [psm-grpc-client-5f9c45c7db-cx2mk:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0912 20:49:34.573943 140449033547776 xds_k8s_testcase.py:389] [psm-grpc-client-5f9c45c7db-cx2mk] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 249
result:
(15, DATA_LOSS): 249

I0912 20:49:34.574714 140449033547776 xds_k8s_testcase.py:272] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0912 20:49:34.575037 140449033547776 xds_k8s_testcase.py:672] ----- TestMethod __main__.CustomLbTest.test_custom_lb_config teardown -----
I0912 20:49:34.811386 140449033547776 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2045-bmll2"
I0912 20:49:35.534951 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551774815-6052f9497655f-ffc94f2e-746e842b
I0912 20:50:03.924304 140449033547776 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2045-bmll2"
I0912 20:50:04.027242 140449033547776 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:04.027473 140449033547776 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2045-bmll2"
I0912 20:50:04.484948 140449033547776 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:04.485180 140449033547776 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2045-bmll2"
I0912 20:50:05.209173 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551804488-6052f965c2bb5-bb878429-71ea202e
I0912 20:50:07.512063 140449033547776 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2045-bmll2"
I0912 20:50:07.848479 140449033547776 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:07.848709 140449033547776 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2045-bmll2"
I0912 20:50:08.618554 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551807854-6052f968f865c-4a3b3815-66e76b51
I0912 20:50:10.971528 140449033547776 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2045-bmll2"
I0912 20:50:11.411267 140449033547776 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:11.411506 140449033547776 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2045-bmll2"
I0912 20:50:12.126089 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551811416-6052f96c5e396-5b69f129-8cc03c96
I0912 20:50:38.391946 140449033547776 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2045-bmll2"
I0912 20:50:38.793823 140449033547776 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:38.794064 140449033547776 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2045-bmll2"
I0912 20:50:39.195131 140449033547776 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2045-bmll2" not deleted since it does not exist
I0912 20:50:39.195364 140449033547776 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2045-bmll2"
I0912 20:50:39.767869 140449033547776 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551839287-6052f986f2908-f0824b0b-50e6d89b
I0912 20:50:44.196335 140449033547776 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 20:51:25.560620 140449033547776 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2045-bmll2/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:51:27.700587 140449033547776 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 20:51:27.821099 140449033547776 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2045-bmll2
I0912 20:51:57.996691 140449033547776 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 20:52:08.178172 140449033547776 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 20:52:08.294471 140449033547776 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2045-bmll2/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:52:09.026740 140449033547776 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 20:52:09.145802 140449033547776 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2045-bmll2
I0912 20:53:09.433230 140449033547776 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 20:53:09.433576 140449033547776 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-y3evaujm:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2045-bmll2%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-y3evaujm%22;timeRange=2023-09-12T20:47:55.117921Z%2F2023-09-12T20:51:57.996642Z?project=grpc-testing
I0912 20:53:09.433713 140449033547776 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-kqfj0jck:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2045-bmll2%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-kqfj0jck%22;timeRange=2023-09-12T20:46:21.430890Z%2F2023-09-12T20:53:09.433049Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0912 20:53:09.433976 140449033547776 base_testcase.py:52] ----- TestCase __main__.CustomLbTest.test_custom_lb_config PASSED -----
----------------------------------------------------------------------
Ran 1 test in 489.310s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/failover_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 20:53:10.803898 140031880019968 xds_k8s_testcase.py:155] ----- Testing FailoverTest -----
I0912 20:53:10.804201 140031880019968 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 20:53:10.804450 140031880019968 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 20:53:10.821625 140031880019968 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 20:53:10.838238 140031880019968 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0912 20:53:10.843021 140031880019968 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2053-h2a6d
I0912 20:53:11.249867 140031880019968 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/failover_test
I0912 20:53:11.567589 140031880019968 xds_k8s_testcase.py:648] Found unused xds port: 60684
I0912 20:53:11.569880 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0912 20:53:11.570023 140031880019968 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2053-h2a6d"
I0912 20:53:11.573093 140031880019968 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2053-h2a6d
type: GRPC
...

I0912 20:53:12.303473 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551991649-6052fa18404bb-a5069924-d9cf6c48
I0912 20:53:14.797210 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0912 20:53:14.797457 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0912 20:53:14.797564 140031880019968 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2053-h2a6d"
I0912 20:53:14.804978 140031880019968 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2053-h2a6d
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2053-h2a6d
protocol: GRPC
...

I0912 20:53:15.504623 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694551994807-6052fa1b4345f-51f72f93-1ef63c78
I0912 20:53:35.344823 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0912 20:53:35.345079 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0912 20:53:35.345179 140031880019968 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2053-h2a6d": psm-grpc-server:60684 -> psm-interop-backend-service-20230912-2053-h2a6d
I0912 20:53:35.356355 140031880019968 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2053-h2a6d
hostRules:
- hosts:
- psm-grpc-server:60684
pathMatcher: psm-interop-path-matcher-20230912-2053-h2a6d
name: psm-interop-url-map-20230912-2053-h2a6d
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2053-h2a6d
name: psm-interop-path-matcher-20230912-2053-h2a6d
...

I0912 20:53:36.037654 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552015358-6052fa2edcbee-2d037e90-40ab9c97
I0912 20:53:38.373913 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0912 20:53:38.374153 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0912 20:53:38.374253 140031880019968 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2053-h2a6d proxy "GRPC" to URL map psm-interop-url-map-20230912-2053-h2a6d
I0912 20:53:38.376169 140031880019968 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2053-h2a6d
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2053-h2a6d
validate_for_proxyless: true
...

I0912 20:53:38.989101 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552018378-6052fa31bdf61-bebe329b-05c2a8d9
I0912 20:53:41.288325 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0912 20:53:41.288554 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0912 20:53:41.288656 140031880019968 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2053-h2a6d" in network "default-vpc": 0.0.0.0:60684 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2053-h2a6d
I0912 20:53:41.291274 140031880019968 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2053-h2a6d
network: global/networks/default-vpc
portRange: 60684
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2053-h2a6d
...

I0912 20:53:42.152494 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552021293-6052fa3485b4d-e319cecb-3639f4a2
I0912 20:53:53.072246 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0912 20:53:53.072483 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0912 20:53:53.072583 140031880019968 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2053-h2a6d: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0912 20:53:53.072851 140031880019968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2053-h2a6d%22;timeRange=2023-09-12T20:53:53.072677Z%2F2023-09-12T21:23:53.072677Z?project=grpc-testing
I0912 20:53:53.077836 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2053-h2a6d
labels:
name: psm-interop-server-20230912-2053-h2a6d
owner: xds-k8s-interop-test
...

I0912 20:53:53.219727 140031880019968 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2053-h2a6d created
I0912 20:53:53.222835 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2053-h2a6d-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:53:53.294854 140031880019968 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 20:53:53.295099 140031880019968 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 20:53:53.392861 140031880019968 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2053-h2a6d-psm-grpc-server in zones=['us-central1-a']
I0912 20:53:53.393076 140031880019968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2053-h2a6d/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:53:54.331142 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:53:54.397088 140031880019968 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 20:53:54.405398 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lj3vat61
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lj3vat61
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lj3vat61
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2053-h2a6d-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:53:54.476419 140031880019968 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 20:53:54.476666 140031880019968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0912 20:53:54.639743 140031880019968 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 20:53:54.640207 140031880019968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-59bc7fccb-5hhkb to start
I0912 20:53:57.875804 140031880019968 k8s_base_runner.py:628] Pod psm-grpc-server-59bc7fccb-5hhkb ready, IP: 10.120.14.245
I0912 20:53:57.876075 140031880019968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-59bc7fccb-5hhkb to failover_test/test_app_logs/psm-interop-server-20230912-2053-h2a6d_psm-grpc-server-59bc7fccb-5hhkb.log
I0912 20:53:57.876513 140031789884992 k8s_log_collector.py:70] Starting log collection thread 140031789884992 for psm-grpc-server-59bc7fccb-5hhkb
I0912 20:53:57.876669 140031880019968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-59bc7fccb-h2r5x to start
I0912 20:53:58.050879 140031880019968 k8s_base_runner.py:628] Pod psm-grpc-server-59bc7fccb-h2r5x ready, IP: 10.120.11.132
I0912 20:53:58.051175 140031880019968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-59bc7fccb-h2r5x to failover_test/test_app_logs/psm-interop-server-20230912-2053-h2a6d_psm-grpc-server-59bc7fccb-h2r5x.log
I0912 20:53:58.051609 140031781492288 k8s_log_collector.py:70] Starting log collection thread 140031781492288 for psm-grpc-server-59bc7fccb-h2r5x
I0912 20:53:58.051860 140031880019968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-59bc7fccb-ztgrr to start
I0912 20:53:58.224418 140031880019968 k8s_base_runner.py:628] Pod psm-grpc-server-59bc7fccb-ztgrr ready, IP: 10.120.9.142
I0912 20:53:58.224783 140031880019968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-59bc7fccb-ztgrr to failover_test/test_app_logs/psm-interop-server-20230912-2053-h2a6d_psm-grpc-server-59bc7fccb-ztgrr.log
I0912 20:53:58.225200 140031704430144 k8s_log_collector.py:70] Starting log collection thread 140031704430144 for psm-grpc-server-59bc7fccb-ztgrr
I0912 20:53:58.225388 140031880019968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0912 20:54:05.600596 140031880019968 k8s_base_runner.py:596] Deployment psm-grpc-server has 3 replicas available
I0912 20:54:05.601205 140031880019968 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230912-2053-h2a6d: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 20:54:05.601460 140031880019968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2053-h2a6d%22;timeRange=2023-09-12T20:54:05.601338Z%2F2023-09-12T21:24:05.601338Z?project=grpc-testing
I0912 20:54:05.603346 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2053-h2a6d
labels:
name: psm-interop-server-20230912-2053-h2a6d
owner: xds-k8s-interop-test
...

I0912 20:54:05.861242 140031880019968 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2053-h2a6d created
I0912 20:54:05.864172 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 20:54:05.977390 140031880019968 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0912 20:54:05.977609 140031880019968 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server-alt
I0912 20:54:06.145114 140031880019968 k8s_base_runner.py:675] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt in zones=['us-west1-b']
I0912 20:54:06.145358 140031880019968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2053-h2a6d/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:54:07.054302 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:54:07.146799 140031880019968 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0912 20:54:07.154247 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2053-h2a6d
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-h0toimzu
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-h0toimzu
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-h0toimzu
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:54:07.260577 140031880019968 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0912 20:54:07.260823 140031880019968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0912 20:54:07.443563 140031880019968 k8s_base_runner.py:615] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 20:54:07.443874 140031880019968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-alt-569d94d48f-zg9zc to start
I0912 20:54:10.848623 140031880019968 k8s_base_runner.py:628] Pod psm-grpc-server-alt-569d94d48f-zg9zc ready, IP: 10.108.11.31
I0912 20:54:10.848940 140031880019968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-alt-569d94d48f-zg9zc to failover_test/test_app_logs/psm-interop-server-20230912-2053-h2a6d_psm-grpc-server-alt-569d94d48f-zg9zc.log
I0912 20:54:10.849362 140031696037440 k8s_log_collector.py:70] Starting log collection thread 140031696037440 for psm-grpc-server-alt-569d94d48f-zg9zc
I0912 20:54:10.849550 140031880019968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0912 20:54:19.621032 140031880019968 k8s_base_runner.py:596] Deployment psm-grpc-server-alt has 1 replicas available
I0912 20:54:19.621445 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0912 20:54:19.621551 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0912 20:54:19.658985 140031880019968 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:54:19.932538 140031880019968 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2053-h2a6d-psm-grpc-server" in zone us-central1-a
I0912 20:54:19.932795 140031880019968 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2053-h2a6d: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2053-h2a6d-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server', zone='us-central1-a')}
I0912 20:54:19.936841 140031880019968 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server
maxRatePerEndpoint: 100
...

I0912 20:54:21.243066 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552059939-6052fa5960aac-1e79b725-b5ff6a38
I0912 20:55:08.851300 140031880019968 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2053-h2a6d to report all backends healthy: ['psm-interop-server-20230912-2053-h2a6d-psm-grpc-server']
I0912 20:55:26.160604 140031880019968 compute.py:487] Backend psm-interop-server-20230912-2053-h2a6d-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:55:30.236972 140031880019968 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 20:55:30.727985 140031880019968 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt" in zone us-west1-b
I0912 20:55:30.728224 140031880019968 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2053-h2a6d: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2053-h2a6d-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt', zone='us-west1-b')}
I0912 20:55:30.733363 140031880019968 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt
maxRatePerEndpoint: 100
...

I0912 20:55:32.452103 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552130736-6052fa9ce4fc9-13a4b886-f17cdca0
I0912 20:56:24.524103 140031880019968 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2053-h2a6d to report all backends healthy: ['psm-interop-server-20230912-2053-h2a6d-psm-grpc-server', 'psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt']
I0912 20:56:24.804379 140031880019968 compute.py:487] Backend psm-interop-server-20230912-2053-h2a6d-psm-grpc-server in zone us-central1-a reported healthy
I0912 20:56:38.621702 140031880019968 compute.py:487] Backend psm-interop-server-20230912-2053-h2a6d-psm-grpc-server-alt in zone us-west1-b reported healthy
I0912 20:56:42.626449 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0912 20:56:42.626704 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0912 20:56:42.626855 140031880019968 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2053-h2a6d: server_target=xds:///psm-grpc-server:60684 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 20:56:42.627115 140031880019968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2053-h2a6d%22;timeRange=2023-09-12T20:56:42.626945Z%2F2023-09-12T21:26:42.626945Z?project=grpc-testing
I0912 20:56:42.629133 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2053-h2a6d
labels:
name: psm-interop-client-20230912-2053-h2a6d
owner: xds-k8s-interop-test
...

I0912 20:56:42.777837 140031880019968 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2053-h2a6d created
I0912 20:56:42.778079 140031880019968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2053-h2a6d/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 20:56:43.463379 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2053-h2a6d
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 20:56:43.524195 140031880019968 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 20:56:43.532569 140031880019968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2053-h2a6d
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q66k3cxu
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q66k3cxu
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q66k3cxu
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:60684"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 20:56:43.607552 140031880019968 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 20:56:43.607820 140031880019968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 20:56:43.707637 140031880019968 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-94d6cd7f5-gsj2q']
I0912 20:56:43.707940 140031880019968 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-94d6cd7f5-gsj2q to start
I0912 20:56:45.898360 140031880019968 k8s_base_runner.py:628] Pod psm-grpc-client-94d6cd7f5-gsj2q ready, IP: 10.120.9.143
I0912 20:56:45.898645 140031880019968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-94d6cd7f5-gsj2q to failover_test/test_app_logs/psm-interop-client-20230912-2053-h2a6d_psm-grpc-client-94d6cd7f5-gsj2q.log
I0912 20:56:45.899094 140031687644736 k8s_log_collector.py:70] Starting log collection thread 140031687644736 for psm-grpc-client-94d6cd7f5-gsj2q
I0912 20:56:45.899248 140031880019968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 20:56:48.085642 140031880019968 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 20:56:48.086272 140031880019968 client_app.py:200] [psm-grpc-client-94d6cd7f5-gsj2q] Waiting to report a READY channel to xds:///psm-grpc-server:60684
I0912 20:56:48.209131 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:56:58.296066 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:57:08.376897 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:57:18.458630 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:57:28.540957 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:57:44.629101 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:58:09.725302 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:58:34.822510 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:58:59.918640 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:59:25.014310 140031880019968 client_app.py:236] [psm-grpc-client-94d6cd7f5-gsj2q] Server channel:
I0912 20:59:25.051684 140031880019968 client_app.py:249] [psm-grpc-client-94d6cd7f5-gsj2q] Found subchannel in state READY:
I0912 20:59:25.051994 140031880019968 client_app.py:211] [psm-grpc-client-94d6cd7f5-gsj2q] Channel to xds:///psm-grpc-server:60684 transitioned to state READY:
I0912 20:59:25.052131 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0912 20:59:25.052210 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0912 20:59:25.052451 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 20:59:25.095326 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0912 20:59:25.095684 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0912 20:59:25.096099 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 20:59:25.096342 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:29.127516 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 33
psm-grpc-server-59bc7fccb-5hhkb: 34
psm-grpc-server-59bc7fccb-ztgrr: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 33
psm-grpc-server-59bc7fccb-5hhkb: 34
psm-grpc-server-59bc7fccb-ztgrr: 33

I0912 20:59:29.128000 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0912 20:59:29.128129 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0912 20:59:29.128214 140031880019968 server_app.py:98] [psm-grpc-server-59bc7fccb-5hhkb] >> Setting health status to NOT_SERVING
I0912 20:59:29.128662 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-5hhkb:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 20:59:29.230932 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-5hhkb:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 20:59:29.264401 140031880019968 server_app.py:102] [psm-grpc-server-59bc7fccb-5hhkb] << Health status status: NOT_SERVING

I0912 20:59:29.264917 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 20:59:29.265148 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:33.301107 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 34
psm-grpc-server-59bc7fccb-5hhkb: 33
psm-grpc-server-59bc7fccb-ztgrr: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 34
psm-grpc-server-59bc7fccb-5hhkb: 33
psm-grpc-server-59bc7fccb-ztgrr: 33

I0912 20:59:33.301842 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-5hhkb' not found in ['psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr'] : Unexpected server psm-grpc-server-59bc7fccb-5hhkb received RPCs.
I0912 20:59:34.303216 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 20:59:34.303594 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:38.317222 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 38
psm-grpc-server-59bc7fccb-5hhkb: 24
psm-grpc-server-59bc7fccb-ztgrr: 38
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 38
psm-grpc-server-59bc7fccb-5hhkb: 24
psm-grpc-server-59bc7fccb-ztgrr: 38

I0912 20:59:38.317735 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-5hhkb' not found in ['psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr'] : Unexpected server psm-grpc-server-59bc7fccb-5hhkb received RPCs.
I0912 20:59:39.319110 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 20:59:39.319488 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:43.337905 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50

I0912 20:59:43.338367 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0912 20:59:43.338553 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0912 20:59:43.338638 140031880019968 server_app.py:98] [psm-grpc-server-59bc7fccb-h2r5x] >> Setting health status to NOT_SERVING
I0912 20:59:43.338969 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-h2r5x:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 20:59:43.440740 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-h2r5x:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 20:59:43.473863 140031880019968 server_app.py:102] [psm-grpc-server-59bc7fccb-h2r5x] << Health status status: NOT_SERVING

I0912 20:59:43.474420 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-ztgrr', 'psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 20:59:43.474633 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:47.507465 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50

I0912 20:59:47.508020 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-569d94d48f-zg9zc' not found in {'psm-grpc-server-59bc7fccb-h2r5x': 50, 'psm-grpc-server-59bc7fccb-ztgrr': 50} : Server psm-grpc-server-alt-569d94d48f-zg9zc did not receive RPCs.
I0912 20:59:48.509605 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-ztgrr', 'psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 20:59:48.509954 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:52.529323 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 50
psm-grpc-server-59bc7fccb-ztgrr: 50

I0912 20:59:52.529901 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-569d94d48f-zg9zc' not found in {'psm-grpc-server-59bc7fccb-h2r5x': 50, 'psm-grpc-server-59bc7fccb-ztgrr': 50} : Server psm-grpc-server-alt-569d94d48f-zg9zc did not receive RPCs.
I0912 20:59:53.531327 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-ztgrr', 'psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 20:59:53.531677 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 20:59:57.575550 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 36
psm-grpc-server-59bc7fccb-ztgrr: 64
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 36
psm-grpc-server-59bc7fccb-ztgrr: 64

I0912 20:59:57.575927 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0912 20:59:57.576110 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0912 20:59:57.576177 140031880019968 server_app.py:98] [psm-grpc-server-59bc7fccb-ztgrr] >> Setting health status to NOT_SERVING
I0912 20:59:57.576476 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-ztgrr:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 20:59:57.677172 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-ztgrr:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 20:59:57.710493 140031880019968 server_app.py:102] [psm-grpc-server-59bc7fccb-ztgrr] << Health status status: NOT_SERVING

I0912 20:59:57.714157 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 20:59:57.714413 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:01.754936 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 28
psm-grpc-server-59bc7fccb-ztgrr: 72
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 28
psm-grpc-server-59bc7fccb-ztgrr: 72

I0912 21:00:01.755782 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-ztgrr' not found in ['psm-grpc-server-alt-569d94d48f-zg9zc'] : Unexpected server psm-grpc-server-59bc7fccb-ztgrr received RPCs.
I0912 21:00:02.757208 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 21:00:02.757555 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:06.769479 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 30
psm-grpc-server-59bc7fccb-ztgrr: 70
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 30
psm-grpc-server-59bc7fccb-ztgrr: 70

I0912 21:00:06.770036 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-ztgrr' not found in ['psm-grpc-server-alt-569d94d48f-zg9zc'] : Unexpected server psm-grpc-server-59bc7fccb-ztgrr received RPCs.
I0912 21:00:07.771484 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-alt-569d94d48f-zg9zc']
I0912 21:00:07.771882 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:11.807437 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 100

I0912 21:00:11.807919 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0912 21:00:11.808056 140031880019968 xds_k8s_testcase.py:267] --- Starting subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0912 21:00:11.808137 140031880019968 server_app.py:89] [psm-grpc-server-59bc7fccb-5hhkb] >> Setting health status to SERVING
I0912 21:00:11.808272 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-5hhkb:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 21:00:11.843723 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-5hhkb:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 21:00:11.877453 140031880019968 server_app.py:91] [psm-grpc-server-59bc7fccb-5hhkb] << Health status status: SERVING

I0912 21:00:11.877688 140031880019968 server_app.py:89] [psm-grpc-server-59bc7fccb-h2r5x] >> Setting health status to SERVING
I0912 21:00:11.877824 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-h2r5x:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 21:00:11.913314 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-h2r5x:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 21:00:11.946756 140031880019968 server_app.py:91] [psm-grpc-server-59bc7fccb-h2r5x] << Health status status: SERVING

I0912 21:00:11.947009 140031880019968 server_app.py:89] [psm-grpc-server-59bc7fccb-ztgrr] >> Setting health status to SERVING
I0912 21:00:11.947148 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-ztgrr:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0912 21:00:11.981845 140031880019968 grpc.py:74] [psm-grpc-server-59bc7fccb-ztgrr:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0912 21:00:12.015439 140031880019968 server_app.py:91] [psm-grpc-server-59bc7fccb-ztgrr] << Health status status: SERVING

I0912 21:00:12.015991 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 21:00:12.016183 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:16.060435 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 100

I0912 21:00:16.060973 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-5hhkb' not found in {'psm-grpc-server-alt-569d94d48f-zg9zc': 100} : Server psm-grpc-server-59bc7fccb-5hhkb did not receive RPCs.
I0912 21:00:17.062613 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 21:00:17.062992 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:21.112253 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-569d94d48f-zg9zc: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-569d94d48f-zg9zc: 100

I0912 21:00:21.112832 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59bc7fccb-5hhkb' not found in {'psm-grpc-server-alt-569d94d48f-zg9zc': 100} : Server psm-grpc-server-59bc7fccb-5hhkb did not receive RPCs.
I0912 21:00:22.114277 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 21:00:22.114623 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:26.135940 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 8
psm-grpc-server-59bc7fccb-5hhkb: 7
psm-grpc-server-alt-569d94d48f-zg9zc: 48
psm-grpc-server-59bc7fccb-ztgrr: 37
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 8
psm-grpc-server-59bc7fccb-5hhkb: 7
psm-grpc-server-alt-569d94d48f-zg9zc: 48
psm-grpc-server-59bc7fccb-ztgrr: 37

I0912 21:00:26.136424 140031880019968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-569d94d48f-zg9zc' not found in ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr'] : Unexpected server psm-grpc-server-alt-569d94d48f-zg9zc received RPCs.
I0912 21:00:27.137779 140031880019968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-59bc7fccb-5hhkb', 'psm-grpc-server-59bc7fccb-h2r5x', 'psm-grpc-server-59bc7fccb-ztgrr']
I0912 21:00:27.138171 140031880019968 grpc.py:74] [psm-grpc-client-94d6cd7f5-gsj2q:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:00:31.152196 140031880019968 xds_k8s_testcase.py:586] [psm-grpc-client-94d6cd7f5-gsj2q] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59bc7fccb-h2r5x: 33
psm-grpc-server-59bc7fccb-5hhkb: 33
psm-grpc-server-59bc7fccb-ztgrr: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-59bc7fccb-h2r5x: 33
psm-grpc-server-59bc7fccb-5hhkb: 33
psm-grpc-server-59bc7fccb-ztgrr: 34

I0912 21:00:31.152735 140031880019968 xds_k8s_testcase.py:272] --- Finished subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0912 21:00:31.152920 140031880019968 xds_k8s_testcase.py:672] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0912 21:00:31.403463 140031880019968 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2053-h2a6d"
I0912 21:00:32.266152 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552431407-6052fbbba326c-b09a3f5c-fcc86c39
I0912 21:00:56.027217 140031880019968 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2053-h2a6d"
I0912 21:00:56.125559 140031880019968 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:00:56.125833 140031880019968 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2053-h2a6d"
I0912 21:00:56.601724 140031880019968 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:00:56.601953 140031880019968 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2053-h2a6d"
I0912 21:00:57.333960 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552456605-6052fbd3aadc9-2b5744c4-f646e728
I0912 21:00:59.644105 140031880019968 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2053-h2a6d"
I0912 21:01:00.042802 140031880019968 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:01:00.043037 140031880019968 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2053-h2a6d"
I0912 21:01:00.816182 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552460048-6052fbd6f3794-7f100851-34a70c37
I0912 21:01:03.072232 140031880019968 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2053-h2a6d"
I0912 21:01:03.517313 140031880019968 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:01:03.517571 140031880019968 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2053-h2a6d"
I0912 21:01:04.208345 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552463523-6052fbda43dde-1921c46c-99ceb938
I0912 21:01:36.833720 140031880019968 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2053-h2a6d"
I0912 21:01:37.297260 140031880019968 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:01:37.297498 140031880019968 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2053-h2a6d"
I0912 21:01:37.817197 140031880019968 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2053-h2a6d" not deleted since it does not exist
I0912 21:01:37.817429 140031880019968 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2053-h2a6d"
I0912 21:01:38.379385 140031880019968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552497897-6052fbfb0c013-d17034e8-65075d74
I0912 21:01:42.915673 140031880019968 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 21:02:23.318500 140031880019968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2053-h2a6d/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:02:23.895066 140031880019968 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 21:02:24.012347 140031880019968 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2053-h2a6d
I0912 21:02:54.239077 140031880019968 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 21:03:04.417329 140031880019968 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 21:03:04.536690 140031880019968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2053-h2a6d/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:03:05.098540 140031880019968 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 21:03:05.218668 140031880019968 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2053-h2a6d
I0912 21:04:05.469666 140031880019968 k8s_base_runner.py:535] Deleting deployment psm-grpc-server-alt
I0912 21:04:15.927635 140031880019968 k8s_base_runner.py:548] Deleting service psm-grpc-server-alt
I0912 21:04:16.127276 140031880019968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2053-h2a6d/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:04:16.816558 140031880019968 k8s_base_runner.py:561] Deleting service account psm-grpc-server-alt
I0912 21:04:17.008943 140031880019968 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2053-h2a6d
I0912 21:06:17.660905 140031880019968 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 21:06:17.661331 140031880019968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-q66k3cxu:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2053-h2a6d%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-q66k3cxu%22;timeRange=2023-09-12T20:56:42.626936Z%2F2023-09-12T21:02:54.239029Z?project=grpc-testing
I0912 21:06:17.661529 140031880019968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-lj3vat61:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2053-h2a6d%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lj3vat61%22;timeRange=2023-09-12T20:53:53.072668Z%2F2023-09-12T21:04:05.469613Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
I0912 21:06:17.661856 140031880019968 base_testcase.py:52] ----- TestCase __main__.FailoverTest.test_failover PASSED -----
----------------------------------------------------------------------
Ran 1 test in 786.861s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/outlier_detection_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 21:06:19.020977 140096603500544 xds_k8s_testcase.py:155] ----- Testing OutlierDetectionTest -----
I0912 21:06:19.021268 140096603500544 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 21:06:19.021566 140096603500544 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 21:06:19.038872 140096603500544 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 21:06:19.056022 140096603500544 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0912 21:06:19.060805 140096603500544 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2106-5fhqq
I0912 21:06:19.490940 140096603500544 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/outlier_detection_test
I0912 21:06:19.866003 140096603500544 xds_k8s_testcase.py:648] Found unused xds port: 11749
I0912 21:06:19.866410 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0912 21:06:19.866523 140096603500544 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2106-5fhqq"
I0912 21:06:19.869733 140096603500544 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2106-5fhqq
type: GRPC
...

I0912 21:06:20.726493 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552779931-6052fd0804034-cbdab11b-f0ff2422
I0912 21:06:23.162085 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0912 21:06:23.162342 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0912 21:06:23.162460 140096603500544 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2106-5fhqq"
I0912 21:06:23.170034 140096603500544 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2106-5fhqq
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2106-5fhqq
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...

I0912 21:06:24.004506 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552783172-6052fd0b1b2c5-12b8322c-2b3f2631
I0912 21:06:43.434658 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0912 21:06:43.434894 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0912 21:06:43.434984 140096603500544 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2106-5fhqq": psm-grpc-server:11749 -> psm-interop-backend-service-20230912-2106-5fhqq
I0912 21:06:43.446172 140096603500544 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2106-5fhqq
hostRules:
- hosts:
- psm-grpc-server:11749
pathMatcher: psm-interop-path-matcher-20230912-2106-5fhqq
name: psm-interop-url-map-20230912-2106-5fhqq
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2106-5fhqq
name: psm-interop-path-matcher-20230912-2106-5fhqq
...

I0912 21:06:44.147921 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552803448-6052fd1e71740-e6b32f17-d87130b2
I0912 21:06:46.433329 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0912 21:06:46.433547 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0912 21:06:46.433644 140096603500544 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2106-5fhqq proxy "GRPC" to URL map psm-interop-url-map-20230912-2106-5fhqq
I0912 21:06:46.435613 140096603500544 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2106-5fhqq
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2106-5fhqq
validate_for_proxyless: true
...

I0912 21:06:47.115828 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552806437-6052fd214b2ea-539460c2-d6cdaa8e
I0912 21:06:49.376511 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0912 21:06:49.376790 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0912 21:06:49.376905 140096603500544 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2106-5fhqq" in network "default-vpc": 0.0.0.0:11749 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2106-5fhqq
I0912 21:06:49.379549 140096603500544 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2106-5fhqq
network: global/networks/default-vpc
portRange: 11749
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2106-5fhqq
...

I0912 21:06:50.181101 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552809381-6052fd2419ef6-c106610b-294e8036
I0912 21:07:22.792695 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0912 21:07:22.792937 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0912 21:07:22.793044 140096603500544 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2106-5fhqq: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0912 21:07:22.793282 140096603500544 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2106-5fhqq%22;timeRange=2023-09-12T21:07:22.793123Z%2F2023-09-12T21:37:22.793123Z?project=grpc-testing
I0912 21:07:22.798249 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2106-5fhqq
labels:
name: psm-interop-server-20230912-2106-5fhqq
owner: xds-k8s-interop-test
...

I0912 21:07:22.949528 140096603500544 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2106-5fhqq created
I0912 21:07:22.952680 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2106-5fhqq
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2106-5fhqq-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 21:07:23.023590 140096603500544 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 21:07:23.023838 140096603500544 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 21:07:23.116216 140096603500544 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2106-5fhqq-psm-grpc-server in zones=['us-central1-a']
I0912 21:07:23.116437 140096603500544 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2106-5fhqq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:07:23.962342 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2106-5fhqq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:07:24.021823 140096603500544 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 21:07:24.030032 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2106-5fhqq
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lvl8bk3f
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lvl8bk3f
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lvl8bk3f
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2106-5fhqq-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:07:24.103009 140096603500544 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 21:07:24.103248 140096603500544 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0912 21:07:24.299548 140096603500544 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:07:24.300060 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-69cfbf6c6f-2k7p8 to start
I0912 21:07:26.558096 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-server-69cfbf6c6f-2k7p8 ready, IP: 10.120.13.218
I0912 21:07:26.558405 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-69cfbf6c6f-2k7p8 to outlier_detection_test/test_app_logs/psm-interop-server-20230912-2106-5fhqq_psm-grpc-server-69cfbf6c6f-2k7p8.log
I0912 21:07:26.558865 140096513238592 k8s_log_collector.py:70] Starting log collection thread 140096513238592 for psm-grpc-server-69cfbf6c6f-2k7p8
I0912 21:07:26.559052 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-69cfbf6c6f-8dsq4 to start
I0912 21:07:26.739185 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-server-69cfbf6c6f-8dsq4 ready, IP: 10.120.9.146
I0912 21:07:26.739503 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-69cfbf6c6f-8dsq4 to outlier_detection_test/test_app_logs/psm-interop-server-20230912-2106-5fhqq_psm-grpc-server-69cfbf6c6f-8dsq4.log
I0912 21:07:26.739994 140096504845888 k8s_log_collector.py:70] Starting log collection thread 140096504845888 for psm-grpc-server-69cfbf6c6f-8dsq4
I0912 21:07:26.740255 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-69cfbf6c6f-m6rrn to start
I0912 21:07:26.918280 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-server-69cfbf6c6f-m6rrn ready, IP: 10.120.17.10
I0912 21:07:26.918575 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-69cfbf6c6f-m6rrn to outlier_detection_test/test_app_logs/psm-interop-server-20230912-2106-5fhqq_psm-grpc-server-69cfbf6c6f-m6rrn.log
I0912 21:07:26.919013 140096496453184 k8s_log_collector.py:70] Starting log collection thread 140096496453184 for psm-grpc-server-69cfbf6c6f-m6rrn
I0912 21:07:26.919157 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-69cfbf6c6f-pv5hd to start
I0912 21:07:27.100294 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-server-69cfbf6c6f-pv5hd ready, IP: 10.120.16.145
I0912 21:07:27.100562 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-69cfbf6c6f-pv5hd to outlier_detection_test/test_app_logs/psm-interop-server-20230912-2106-5fhqq_psm-grpc-server-69cfbf6c6f-pv5hd.log
I0912 21:07:27.100979 140096488060480 k8s_log_collector.py:70] Starting log collection thread 140096488060480 for psm-grpc-server-69cfbf6c6f-pv5hd
I0912 21:07:27.101123 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-69cfbf6c6f-wwvsd to start
I0912 21:07:27.280108 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-server-69cfbf6c6f-wwvsd ready, IP: 10.120.10.225
I0912 21:07:27.280389 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-69cfbf6c6f-wwvsd to outlier_detection_test/test_app_logs/psm-interop-server-20230912-2106-5fhqq_psm-grpc-server-69cfbf6c6f-wwvsd.log
I0912 21:07:27.280807 140096479667776 k8s_log_collector.py:70] Starting log collection thread 140096479667776 for psm-grpc-server-69cfbf6c6f-wwvsd
I0912 21:07:27.280951 140096603500544 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0912 21:07:33.689157 140096603500544 k8s_base_runner.py:596] Deployment psm-grpc-server has 5 replicas available
I0912 21:07:33.689931 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0912 21:07:33.690085 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0912 21:07:33.726051 140096603500544 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 21:07:34.053820 140096603500544 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2106-5fhqq-psm-grpc-server" in zone us-central1-a
I0912 21:07:34.054052 140096603500544 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2106-5fhqq: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2106-5fhqq-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2106-5fhqq-psm-grpc-server', zone='us-central1-a')}
I0912 21:07:34.058063 140096603500544 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2106-5fhqq-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 21:07:35.278590 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694552854060-6052fd4eb5f13-ee193ad0-cb1a39d4
I0912 21:08:18.930975 140096603500544 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2106-5fhqq to report all backends healthy: ['psm-interop-server-20230912-2106-5fhqq-psm-grpc-server']
I0912 21:08:27.987855 140096603500544 compute.py:487] Backend psm-interop-server-20230912-2106-5fhqq-psm-grpc-server in zone us-central1-a reported healthy
I0912 21:08:31.992541 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0912 21:08:31.992798 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0912 21:08:31.992918 140096603500544 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2106-5fhqq: server_target=xds:///psm-grpc-server:11749 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0912 21:08:31.993140 140096603500544 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2106-5fhqq%22;timeRange=2023-09-12T21:08:31.993003Z%2F2023-09-12T21:38:31.993003Z?project=grpc-testing
I0912 21:08:31.995163 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2106-5fhqq
labels:
name: psm-interop-client-20230912-2106-5fhqq
owner: xds-k8s-interop-test
...

I0912 21:08:32.036563 140096603500544 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2106-5fhqq created
I0912 21:08:32.036803 140096603500544 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2106-5fhqq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:08:32.660013 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2106-5fhqq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:08:32.722639 140096603500544 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 21:08:32.730432 140096603500544 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2106-5fhqq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nusdj7t8
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nusdj7t8
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-nusdj7t8
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:11749"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:08:32.805544 140096603500544 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 21:08:32.805805 140096603500544 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 21:08:32.915276 140096603500544 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-677d875dc7-pgzb6']
I0912 21:08:32.915535 140096603500544 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-677d875dc7-pgzb6 to start
I0912 21:08:35.108206 140096603500544 k8s_base_runner.py:628] Pod psm-grpc-client-677d875dc7-pgzb6 ready, IP: 10.120.14.249
I0912 21:08:35.108475 140096603500544 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-677d875dc7-pgzb6 to outlier_detection_test/test_app_logs/psm-interop-client-20230912-2106-5fhqq_psm-grpc-client-677d875dc7-pgzb6.log
I0912 21:08:35.108904 140096128939584 k8s_log_collector.py:70] Starting log collection thread 140096128939584 for psm-grpc-client-677d875dc7-pgzb6
I0912 21:08:35.109048 140096603500544 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 21:08:37.366533 140096603500544 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 21:08:37.367118 140096603500544 client_app.py:200] [psm-grpc-client-677d875dc7-pgzb6] Waiting to report a READY channel to xds:///psm-grpc-server:11749
I0912 21:08:37.486706 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:08:47.569733 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:08:57.649685 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:09:07.730683 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:09:17.812695 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:09:33.900463 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:09:59.007740 140096603500544 client_app.py:236] [psm-grpc-client-677d875dc7-pgzb6] Server channel:
I0912 21:09:59.047533 140096603500544 client_app.py:249] [psm-grpc-client-677d875dc7-pgzb6] Found subchannel in state READY:
I0912 21:09:59.048019 140096603500544 client_app.py:211] [psm-grpc-client-677d875dc7-pgzb6] Channel to xds:///psm-grpc-server:11749 transitioned to state READY:
I0912 21:09:59.048187 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0912 21:09:59.048274 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0912 21:09:59.048488 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0912 21:09:59.092656 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0912 21:09:59.092985 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0912 21:09:59.093335 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:09:59.093564 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:00.128667 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 20
psm-grpc-server-69cfbf6c6f-wwvsd: 20
psm-grpc-server-69cfbf6c6f-8dsq4: 20
psm-grpc-server-69cfbf6c6f-pv5hd: 20
psm-grpc-server-69cfbf6c6f-2k7p8: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 20
psm-grpc-server-69cfbf6c6f-wwvsd: 20
psm-grpc-server-69cfbf6c6f-8dsq4: 20
psm-grpc-server-69cfbf6c6f-pv5hd: 20
psm-grpc-server-69cfbf6c6f-2k7p8: 20

I0912 21:10:00.129250 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0912 21:10:00.129392 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0912 21:10:00.129723 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-69cfbf6c6f-2k7p8 error-code-2'}]}), wait_for_ready=True, timeout=5)
I0912 21:10:00.167719 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:00.168106 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:01.206832 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 13
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 22
psm-grpc-server-69cfbf6c6f-m6rrn: 21
psm-grpc-server-69cfbf6c6f-8dsq4: 22
psm-grpc-server-69cfbf6c6f-pv5hd: 22
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 22
psm-grpc-server-69cfbf6c6f-m6rrn: 21
psm-grpc-server-69cfbf6c6f-8dsq4: 22
psm-grpc-server-69cfbf6c6f-pv5hd: 22

I0912 21:10:01.207633 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 13 not less than or equal to 0 : Expected all RPCs to succeed: 13 of 100 failed.
I0912 21:10:02.209052 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:02.209392 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:03.243808 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:03.244172 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0912 21:10:03.244449 140096603500544 xds_k8s_testcase.py:267] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0912 21:10:03.244662 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), wait_for_ready=True, timeout=5)
I0912 21:10:03.279445 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:03.279790 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:04.320457 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:04.321135 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:05.322444 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:05.322818 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:06.360193 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:06.360718 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:07.362175 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:07.362513 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:08.408877 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:08.409425 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:09.410707 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:09.411090 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:10.448268 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:10.448811 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:11.450197 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:11.450548 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:12.494037 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:12.494604 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:13.495917 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:13.496249 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:14.535948 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:14.536489 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:15.537840 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:15.538240 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:16.576173 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:16.576665 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:17.578022 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:17.578388 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:18.615139 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:18.615658 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:19.616989 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:19.617343 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:20.659305 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:20.659677 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:21.661092 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:21.661438 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:22.686255 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:22.686716 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:23.688049 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:23.688395 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:24.725162 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:24.725678 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:25.726965 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:25.727335 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:26.770207 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:26.770675 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:27.772114 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:27.772474 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:28.803714 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:28.804190 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:29.805703 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:29.806070 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:30.847149 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-m6rrn: 25
psm-grpc-server-69cfbf6c6f-wwvsd: 25
psm-grpc-server-69cfbf6c6f-8dsq4: 25
psm-grpc-server-69cfbf6c6f-pv5hd: 25

I0912 21:10:30.847656 140096603500544 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cfbf6c6f-2k7p8' not found in {'psm-grpc-server-69cfbf6c6f-m6rrn': 25, 'psm-grpc-server-69cfbf6c6f-wwvsd': 25, 'psm-grpc-server-69cfbf6c6f-8dsq4': 25, 'psm-grpc-server-69cfbf6c6f-pv5hd': 25} : Server psm-grpc-server-69cfbf6c6f-2k7p8 did not receive RPCs.
I0912 21:10:31.849025 140096603500544 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-69cfbf6c6f-2k7p8', 'psm-grpc-server-69cfbf6c6f-8dsq4', 'psm-grpc-server-69cfbf6c6f-m6rrn', 'psm-grpc-server-69cfbf6c6f-pv5hd', 'psm-grpc-server-69cfbf6c6f-wwvsd']
I0912 21:10:31.849355 140096603500544 grpc.py:74] [psm-grpc-client-677d875dc7-pgzb6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:10:32.885792 140096603500544 xds_k8s_testcase.py:586] [psm-grpc-client-677d875dc7-pgzb6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cfbf6c6f-wwvsd: 20
psm-grpc-server-69cfbf6c6f-m6rrn: 20
psm-grpc-server-69cfbf6c6f-8dsq4: 20
psm-grpc-server-69cfbf6c6f-pv5hd: 20
psm-grpc-server-69cfbf6c6f-2k7p8: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cfbf6c6f-wwvsd: 20
psm-grpc-server-69cfbf6c6f-m6rrn: 20
psm-grpc-server-69cfbf6c6f-8dsq4: 20
psm-grpc-server-69cfbf6c6f-pv5hd: 20
psm-grpc-server-69cfbf6c6f-2k7p8: 20

I0912 21:10:32.886227 140096603500544 xds_k8s_testcase.py:272] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0912 21:10:32.886378 140096603500544 xds_k8s_testcase.py:672] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0912 21:10:33.167760 140096603500544 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2106-5fhqq"
I0912 21:10:34.001537 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553033172-6052fdf986453-16d304aa-215b3a5c
I0912 21:10:42.823796 140096603500544 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2106-5fhqq"
I0912 21:10:42.949089 140096603500544 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:10:42.949330 140096603500544 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2106-5fhqq"
I0912 21:10:43.365990 140096603500544 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:10:43.366245 140096603500544 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2106-5fhqq"
I0912 21:10:44.000151 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553043369-6052fe033fded-34954f94-8caf2734
I0912 21:10:46.385833 140096603500544 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2106-5fhqq"
I0912 21:10:46.873829 140096603500544 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:10:46.874061 140096603500544 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2106-5fhqq"
I0912 21:10:47.548918 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553046879-6052fe0698e12-679ffc1e-bc835236
I0912 21:10:49.881240 140096603500544 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2106-5fhqq"
I0912 21:10:50.289141 140096603500544 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:10:50.289372 140096603500544 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2106-5fhqq"
I0912 21:10:51.002519 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553050295-6052fe09dabeb-71b25fd1-176f712d
I0912 21:11:19.437510 140096603500544 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2106-5fhqq"
I0912 21:11:19.901021 140096603500544 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:11:19.901257 140096603500544 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2106-5fhqq"
I0912 21:11:20.384880 140096603500544 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2106-5fhqq" not deleted since it does not exist
I0912 21:11:20.385109 140096603500544 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2106-5fhqq"
I0912 21:11:20.923729 140096603500544 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553080470-6052fe26a1be5-bb72537c-d688cf0d
I0912 21:11:25.535199 140096603500544 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 21:12:05.952423 140096603500544 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2106-5fhqq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:12:06.562595 140096603500544 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 21:12:06.684478 140096603500544 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2106-5fhqq
I0912 21:12:36.921847 140096603500544 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 21:12:47.117819 140096603500544 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 21:12:47.238799 140096603500544 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2106-5fhqq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:12:47.953870 140096603500544 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 21:12:48.079619 140096603500544 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2106-5fhqq
I0912 21:15:18.579123 140096603500544 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 21:15:18.579495 140096603500544 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-nusdj7t8:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2106-5fhqq%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-nusdj7t8%22;timeRange=2023-09-12T21:08:31.992994Z%2F2023-09-12T21:12:36.921798Z?project=grpc-testing
I0912 21:15:18.579638 140096603500544 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-lvl8bk3f:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2106-5fhqq%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lvl8bk3f%22;timeRange=2023-09-12T21:07:22.793116Z%2F2023-09-12T21:15:18.578938Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
I0912 21:15:18.579898 140096603500544 base_testcase.py:52] ----- TestCase __main__.OutlierDetectionTest.test_outlier_detection PASSED -----
----------------------------------------------------------------------
Ran 1 test in 539.562s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/remove_neg_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 21:15:19.929704 140216836435968 xds_k8s_testcase.py:155] ----- Testing RemoveNegTest -----
I0912 21:15:19.929854 140216836435968 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 21:15:19.930416 140216836435968 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 21:15:19.947502 140216836435968 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 21:15:19.965317 140216836435968 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0912 21:15:19.970093 140216836435968 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2115-6360u
I0912 21:15:20.378304 140216836435968 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/remove_neg_test
I0912 21:15:20.786084 140216836435968 xds_k8s_testcase.py:648] Found unused xds port: 25806
I0912 21:15:20.788550 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0912 21:15:20.788758 140216836435968 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2115-6360u"
I0912 21:15:20.791851 140216836435968 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2115-6360u
type: GRPC
...

I0912 21:15:21.487990 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553320876-6052ff0be68c1-1b50b66a-cd19a93e
I0912 21:15:23.980321 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0912 21:15:23.980582 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0912 21:15:23.980693 140216836435968 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2115-6360u"
I0912 21:15:23.987774 140216836435968 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2115-6360u
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2115-6360u
protocol: GRPC
...

I0912 21:15:24.762460 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553323990-6052ff0edec93-14e6664f-c42bb5fb
I0912 21:15:46.606846 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0912 21:15:46.607081 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0912 21:15:46.607190 140216836435968 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2115-6360u": psm-grpc-server:25806 -> psm-interop-backend-service-20230912-2115-6360u
I0912 21:15:46.618319 140216836435968 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2115-6360u
hostRules:
- hosts:
- psm-grpc-server:25806
pathMatcher: psm-interop-path-matcher-20230912-2115-6360u
name: psm-interop-url-map-20230912-2115-6360u
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2115-6360u
name: psm-interop-path-matcher-20230912-2115-6360u
...

I0912 21:15:47.345609 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553346620-6052ff2473c76-ce2cab07-a4e1c594
I0912 21:15:49.696462 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0912 21:15:49.696682 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0912 21:15:49.696793 140216836435968 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2115-6360u proxy "GRPC" to URL map psm-interop-url-map-20230912-2115-6360u
I0912 21:15:49.698757 140216836435968 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2115-6360u
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2115-6360u
validate_for_proxyless: true
...

I0912 21:15:50.139475 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553349701-6052ff2763e20-d5c86149-f690ac36
I0912 21:15:52.415626 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0912 21:15:52.415861 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0912 21:15:52.415966 140216836435968 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2115-6360u" in network "default-vpc": 0.0.0.0:25806 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2115-6360u
I0912 21:15:52.418578 140216836435968 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2115-6360u
network: global/networks/default-vpc
portRange: 25806
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2115-6360u
...

I0912 21:15:53.242023 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553352420-6052ff29fbdae-bdd19864-12667af3
I0912 21:16:06.478912 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0912 21:16:06.479165 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0912 21:16:06.479268 140216836435968 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2115-6360u: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 21:16:06.479523 140216836435968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2115-6360u%22;timeRange=2023-09-12T21:16:06.479357Z%2F2023-09-12T21:46:06.479357Z?project=grpc-testing
I0912 21:16:06.484874 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2115-6360u
labels:
name: psm-interop-server-20230912-2115-6360u
owner: xds-k8s-interop-test
...

I0912 21:16:06.630278 140216836435968 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2115-6360u created
I0912 21:16:06.633170 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2115-6360u
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2115-6360u-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 21:16:06.714278 140216836435968 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 21:16:06.714499 140216836435968 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 21:16:06.829961 140216836435968 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2115-6360u-psm-grpc-server in zones=['us-central1-a']
I0912 21:16:06.830217 140216836435968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2115-6360u/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:16:07.619218 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2115-6360u
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:16:07.690316 140216836435968 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 21:16:07.698628 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2115-6360u
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-uz5dbbi2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-uz5dbbi2
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-uz5dbbi2
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2115-6360u-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:16:07.780414 140216836435968 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 21:16:07.780650 140216836435968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0912 21:16:07.896757 140216836435968 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-7799458fd8-64lsk']
I0912 21:16:07.897028 140216836435968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-7799458fd8-64lsk to start
I0912 21:16:11.170129 140216836435968 k8s_base_runner.py:628] Pod psm-grpc-server-7799458fd8-64lsk ready, IP: 10.120.11.137
I0912 21:16:11.170391 140216836435968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-7799458fd8-64lsk to remove_neg_test/test_app_logs/psm-interop-server-20230912-2115-6360u_psm-grpc-server-7799458fd8-64lsk.log
I0912 21:16:11.170811 140216746108480 k8s_log_collector.py:70] Starting log collection thread 140216746108480 for psm-grpc-server-7799458fd8-64lsk
I0912 21:16:11.170961 140216836435968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0912 21:16:16.590862 140216836435968 k8s_base_runner.py:596] Deployment psm-grpc-server has 1 replicas available
I0912 21:16:16.591319 140216836435968 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230912-2115-6360u: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0912 21:16:16.591543 140216836435968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2115-6360u%22;timeRange=2023-09-12T21:16:16.591426Z%2F2023-09-12T21:46:16.591426Z?project=grpc-testing
I0912 21:16:16.591622 140216836435968 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20230912-2115-6360u
I0912 21:16:16.657880 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2115-6360u
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 21:16:16.708315 140216836435968 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0912 21:16:16.708551 140216836435968 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server-alt
I0912 21:16:16.812409 140216836435968 k8s_base_runner.py:675] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt in zones=['us-central1-a']
I0912 21:16:16.812663 140216836435968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2115-6360u/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:16:17.534167 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2115-6360u
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:16:17.583102 140216836435968 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0912 21:16:17.591036 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230912-2115-6360u
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-yu8awmsf
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-yu8awmsf
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-yu8awmsf
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:16:17.648886 140216836435968 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0912 21:16:17.649138 140216836435968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0912 21:16:17.751129 140216836435968 k8s_base_runner.py:615] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-d9cb7645c-h8p7r']
I0912 21:16:17.751402 140216836435968 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-alt-d9cb7645c-h8p7r to start
I0912 21:16:21.088530 140216836435968 k8s_base_runner.py:628] Pod psm-grpc-server-alt-d9cb7645c-h8p7r ready, IP: 10.120.17.11
I0912 21:16:21.088809 140216836435968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-alt-d9cb7645c-h8p7r to remove_neg_test/test_app_logs/psm-interop-server-20230912-2115-6360u_psm-grpc-server-alt-d9cb7645c-h8p7r.log
I0912 21:16:21.089205 140216737715776 k8s_log_collector.py:70] Starting log collection thread 140216737715776 for psm-grpc-server-alt-d9cb7645c-h8p7r
I0912 21:16:21.089361 140216836435968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0912 21:16:27.484704 140216836435968 k8s_base_runner.py:596] Deployment psm-grpc-server-alt has 1 replicas available
I0912 21:16:27.485137 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0912 21:16:27.485251 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0912 21:16:27.528390 140216836435968 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 21:16:27.803476 140216836435968 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2115-6360u-psm-grpc-server" in zone us-central1-a
I0912 21:16:27.803775 140216836435968 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2115-6360u: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2115-6360u-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server', zone='us-central1-a')}
I0912 21:16:27.807963 140216836435968 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 21:16:28.703979 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553387810-6052ff4bbbe79-cf5f2bd4-32e0342e
I0912 21:17:18.698503 140216836435968 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2115-6360u to report all backends healthy: ['psm-interop-server-20230912-2115-6360u-psm-grpc-server']
I0912 21:17:31.907935 140216836435968 compute.py:487] Backend psm-interop-server-20230912-2115-6360u-psm-grpc-server in zone us-central1-a reported healthy
I0912 21:17:36.014444 140216836435968 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 21:17:36.314091 140216836435968 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt" in zone us-central1-a
I0912 21:17:36.314340 140216836435968 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2115-6360u: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2115-6360u-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt', zone='us-central1-a')}
I0912 21:17:36.318906 140216836435968 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0912 21:17:37.403215 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553456321-6052ff8d12239-89ef1277-747b9382
I0912 21:18:31.565971 140216836435968 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2115-6360u to report all backends healthy: ['psm-interop-server-20230912-2115-6360u-psm-grpc-server', 'psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt']
I0912 21:18:31.885002 140216836435968 compute.py:487] Backend psm-interop-server-20230912-2115-6360u-psm-grpc-server in zone us-central1-a reported healthy
I0912 21:18:36.100104 140216836435968 compute.py:487] Backend psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt in zone us-central1-a reported healthy
I0912 21:18:40.104973 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0912 21:18:40.105211 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0912 21:18:40.105326 140216836435968 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2115-6360u: server_target=xds:///psm-grpc-server:25806 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 21:18:40.105583 140216836435968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2115-6360u%22;timeRange=2023-09-12T21:18:40.105414Z%2F2023-09-12T21:48:40.105414Z?project=grpc-testing
I0912 21:18:40.107836 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2115-6360u
labels:
name: psm-interop-client-20230912-2115-6360u
owner: xds-k8s-interop-test
...

I0912 21:18:40.183567 140216836435968 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2115-6360u created
I0912 21:18:40.183825 140216836435968 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2115-6360u/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:18:40.892604 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2115-6360u
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:18:40.959589 140216836435968 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 21:18:40.967338 140216836435968 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2115-6360u
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-opav9uni
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-opav9uni
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-opav9uni
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:25806"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:18:41.053520 140216836435968 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 21:18:41.053809 140216836435968 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 21:18:41.177782 140216836435968 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8599669664-hn89d']
I0912 21:18:41.178046 140216836435968 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-8599669664-hn89d to start
I0912 21:18:44.456217 140216836435968 k8s_base_runner.py:628] Pod psm-grpc-client-8599669664-hn89d ready, IP: 10.120.10.227
I0912 21:18:44.456485 140216836435968 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-8599669664-hn89d to remove_neg_test/test_app_logs/psm-interop-client-20230912-2115-6360u_psm-grpc-client-8599669664-hn89d.log
I0912 21:18:44.456921 140216656459328 k8s_log_collector.py:70] Starting log collection thread 140216656459328 for psm-grpc-client-8599669664-hn89d
I0912 21:18:44.457064 140216836435968 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 21:18:47.783163 140216836435968 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 21:18:47.783739 140216836435968 client_app.py:200] [psm-grpc-client-8599669664-hn89d] Waiting to report a READY channel to xds:///psm-grpc-server:25806
I0912 21:18:47.906053 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:18:57.993471 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:19:08.077126 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:19:18.159608 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:19:28.243075 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:19:44.332086 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:20:09.431300 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:20:34.529342 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:20:59.626956 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:21:24.724938 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:21:49.821633 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:22:14.919623 140216836435968 client_app.py:236] [psm-grpc-client-8599669664-hn89d] Server channel:
I0912 21:22:14.958971 140216836435968 client_app.py:249] [psm-grpc-client-8599669664-hn89d] Found subchannel in state READY:
I0912 21:22:14.959314 140216836435968 client_app.py:211] [psm-grpc-client-8599669664-hn89d] Channel to xds:///psm-grpc-server:25806 transitioned to state READY:
I0912 21:22:14.959549 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0912 21:22:14.959653 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0912 21:22:14.959917 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 21:22:15.004918 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0912 21:22:15.005244 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0912 21:22:15.005488 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:22:19.050385 140216836435968 xds_k8s_testcase.py:586] [psm-grpc-client-8599669664-hn89d] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7799458fd8-64lsk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7799458fd8-64lsk: 100

I0912 21:22:19.050675 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0912 21:22:19.050766 140216836435968 xds_k8s_testcase.py:267] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0912 21:22:19.051128 140216836435968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-7799458fd8-64lsk', 'psm-grpc-server-alt-d9cb7645c-h8p7r']
I0912 21:22:19.051285 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:22:23.156030 140216836435968 xds_k8s_testcase.py:586] [psm-grpc-client-8599669664-hn89d] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7799458fd8-64lsk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7799458fd8-64lsk: 100

I0912 21:22:23.156790 140216836435968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-d9cb7645c-h8p7r' not found in {'psm-grpc-server-7799458fd8-64lsk': 100} : Server psm-grpc-server-alt-d9cb7645c-h8p7r did not receive RPCs.
I0912 21:22:24.158211 140216836435968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-7799458fd8-64lsk', 'psm-grpc-server-alt-d9cb7645c-h8p7r']
I0912 21:22:24.158561 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:22:28.201127 140216836435968 xds_k8s_testcase.py:586] [psm-grpc-client-8599669664-hn89d] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7799458fd8-64lsk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7799458fd8-64lsk: 100

I0912 21:22:28.201675 140216836435968 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-d9cb7645c-h8p7r' not found in {'psm-grpc-server-7799458fd8-64lsk': 100} : Server psm-grpc-server-alt-d9cb7645c-h8p7r did not receive RPCs.
I0912 21:22:29.202962 140216836435968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-7799458fd8-64lsk', 'psm-grpc-server-alt-d9cb7645c-h8p7r']
I0912 21:22:29.203323 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:22:33.284203 140216836435968 xds_k8s_testcase.py:586] [psm-grpc-client-8599669664-hn89d] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-d9cb7645c-h8p7r: 59
psm-grpc-server-7799458fd8-64lsk: 41
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-d9cb7645c-h8p7r: 59
psm-grpc-server-7799458fd8-64lsk: 41

I0912 21:22:34.401234 140216836435968 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0912 21:22:34.695092 140216836435968 traffic_director.py:264] Loaded NEG "psm-interop-server-20230912-2115-6360u-psm-grpc-server-alt" in zone us-central1-a
I0912 21:22:34.695337 140216836435968 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2115-6360u: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2115-6360u-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server', zone='us-central1-a')}
I0912 21:22:34.699560 140216836435968 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2115-6360u-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 21:22:35.655033 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553754701-605300a9a10a9-6fbbe04d-2de3d334
I0912 21:23:08.550275 140216836435968 xds_k8s_testcase.py:463] Verifying RPCs go to servers ['psm-grpc-server-7799458fd8-64lsk']
I0912 21:23:08.550593 140216836435968 grpc.py:74] [psm-grpc-client-8599669664-hn89d:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0912 21:23:12.630598 140216836435968 xds_k8s_testcase.py:586] [psm-grpc-client-8599669664-hn89d] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7799458fd8-64lsk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7799458fd8-64lsk: 100

I0912 21:23:12.631037 140216836435968 xds_k8s_testcase.py:272] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0912 21:23:12.631206 140216836435968 xds_k8s_testcase.py:672] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0912 21:23:12.761571 140216836435968 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2115-6360u"
I0912 21:23:13.687808 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553792765-605300cdedc97-8eea18e8-5098d7ed
I0912 21:23:20.453849 140216836435968 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2115-6360u"
I0912 21:23:20.566523 140216836435968 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:20.566793 140216836435968 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2115-6360u"
I0912 21:23:21.001851 140216836435968 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:21.002124 140216836435968 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2115-6360u"
I0912 21:23:21.422229 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553801005-605300d5c98e2-7cdc0227-5c748e63
I0912 21:23:23.767992 140216836435968 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2115-6360u"
I0912 21:23:24.026142 140216836435968 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:24.026389 140216836435968 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2115-6360u"
I0912 21:23:24.690842 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553804031-605300d8ac6e8-fed0c295-a1725df8
I0912 21:23:26.999553 140216836435968 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2115-6360u"
I0912 21:23:27.482361 140216836435968 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:27.482597 140216836435968 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2115-6360u"
I0912 21:23:28.162151 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553807487-605300dbf8211-f735462e-341038e5
I0912 21:23:58.641666 140216836435968 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2115-6360u"
I0912 21:23:59.099960 140216836435968 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:59.100195 140216836435968 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2115-6360u"
I0912 21:23:59.605250 140216836435968 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2115-6360u" not deleted since it does not exist
I0912 21:23:59.605484 140216836435968 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2115-6360u"
I0912 21:24:00.186543 140216836435968 compute.py:589] Waiting 600 sec for compute operation id: operation-1694553839718-605300fab5117-bba23198-443594f5
I0912 21:24:04.668979 140216836435968 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 21:24:45.120914 140216836435968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2115-6360u/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:24:45.819571 140216836435968 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 21:24:45.943946 140216836435968 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2115-6360u
I0912 21:25:16.111587 140216836435968 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 21:25:26.293352 140216836435968 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 21:25:26.416310 140216836435968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2115-6360u/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:25:27.122248 140216836435968 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 21:25:27.247391 140216836435968 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2115-6360u
I0912 21:27:27.657721 140216836435968 k8s_base_runner.py:535] Deleting deployment psm-grpc-server-alt
I0912 21:27:27.723562 140216836435968 k8s_base_runner.py:540] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0912 21:27:27.723903 140216836435968 k8s_base_runner.py:548] Deleting service psm-grpc-server-alt
I0912 21:27:27.781371 140216836435968 k8s_base_runner.py:552] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0912 21:27:27.781693 140216836435968 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2115-6360u/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:27:28.515240 140216836435968 k8s_base_runner.py:561] Deleting service account psm-grpc-server-alt
I0912 21:27:28.572999 140216836435968 k8s_base_runner.py:565] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0912 21:27:28.573277 140216836435968 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2115-6360u
I0912 21:27:28.630163 140216836435968 k8s_base_runner.py:577] Namespace psm-interop-server-20230912-2115-6360u deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230912-2115-6360u" not found
I0912 21:27:28.630685 140216836435968 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 21:27:28.630977 140216836435968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-opav9uni:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2115-6360u%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-opav9uni%22;timeRange=2023-09-12T21:18:40.105403Z%2F2023-09-12T21:25:16.111533Z?project=grpc-testing
I0912 21:27:28.631129 140216836435968 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-uz5dbbi2:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2115-6360u%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-uz5dbbi2%22;timeRange=2023-09-12T21:16:06.479347Z%2F2023-09-12T21:27:27.657672Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
I0912 21:27:28.631422 140216836435968 base_testcase.py:52] ----- TestCase __main__.RemoveNegTest.test_remove_neg PASSED -----
----------------------------------------------------------------------
Ran 1 test in 728.705s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/round_robin_test'
Running tests under Python 3.10.12: /tmp/tmp.WKlxHh40Vl/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python3
I0912 21:27:30.070898 140512254185472 xds_k8s_testcase.py:155] ----- Testing RoundRobinTest -----
I0912 21:27:30.071060 140512254185472 xds_k8s_testcase.py:156] Logs timezone: UTC
I0912 21:27:30.071422 140512254185472 xds_k8s_testcase.py:104] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0912 21:27:30.088676 140512254185472 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0912 21:27:30.106269 140512254185472 k8s.py:184] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0912 21:27:30.111160 140512254185472 xds_k8s_testcase.py:653] Test run resource prefix: psm-interop, suffix: 20230912-2127-11m4e
I0912 21:27:30.501050 140512254185472 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/round_robin_test
I0912 21:27:30.896792 140512254185472 xds_k8s_testcase.py:648] Found unused xds port: 27043
I0912 21:27:30.897140 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0912 21:27:30.897223 140512254185472 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230912-2127-11m4e"
I0912 21:27:30.900347 140512254185472 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230912-2127-11m4e
type: GRPC
...

I0912 21:27:31.632836 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554050977-605301c42ddb4-e1d3aa1c-a6c948c6
I0912 21:27:33.968947 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0912 21:27:33.969179 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0912 21:27:33.969290 140512254185472 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230912-2127-11m4e"
I0912 21:27:33.976695 140512254185472 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230912-2127-11m4e
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230912-2127-11m4e
protocol: GRPC
...

I0912 21:27:34.804156 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554053978-605301c70ab01-fb0c74b9-315e616b
I0912 21:27:54.564054 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0912 21:27:54.564307 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0912 21:27:54.564406 140512254185472 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230912-2127-11m4e": psm-grpc-server:27043 -> psm-interop-backend-service-20230912-2127-11m4e
I0912 21:27:54.575578 140512254185472 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2127-11m4e
hostRules:
- hosts:
- psm-grpc-server:27043
pathMatcher: psm-interop-path-matcher-20230912-2127-11m4e
name: psm-interop-url-map-20230912-2127-11m4e
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230912-2127-11m4e
name: psm-interop-path-matcher-20230912-2127-11m4e
...

I0912 21:27:55.288821 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554074578-605301daafe54-a155fdd0-7009c5bf
I0912 21:27:57.647691 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0912 21:27:57.647926 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0912 21:27:57.648027 140512254185472 traffic_director.py:574] Creating target psm-interop-target-proxy-20230912-2127-11m4e proxy "GRPC" to URL map psm-interop-url-map-20230912-2127-11m4e
I0912 21:27:57.650032 140512254185472 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230912-2127-11m4e
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230912-2127-11m4e
validate_for_proxyless: true
...

I0912 21:27:58.374828 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554077652-605301dd9e52f-13499e4c-cf137689
I0912 21:28:00.752840 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0912 21:28:00.753066 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0912 21:28:00.753174 140512254185472 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230912-2127-11m4e" in network "default-vpc": 0.0.0.0:27043 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2127-11m4e
I0912 21:28:00.755866 140512254185472 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230912-2127-11m4e
network: global/networks/default-vpc
portRange: 27043
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230912-2127-11m4e
...

I0912 21:28:01.538379 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554080757-605301e094852-117979fd-7589c5a5
I0912 21:28:12.490966 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0912 21:28:12.491209 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0912 21:28:12.491315 140512254185472 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230912-2127-11m4e: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0912 21:28:12.491578 140512254185472 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2127-11m4e%22;timeRange=2023-09-12T21:28:12.491422Z%2F2023-09-12T21:58:12.491422Z?project=grpc-testing
I0912 21:28:12.496531 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230912-2127-11m4e
labels:
name: psm-interop-server-20230912-2127-11m4e
owner: xds-k8s-interop-test
...

I0912 21:28:12.631378 140512254185472 k8s_base_runner.py:284] Namespace psm-interop-server-20230912-2127-11m4e created
I0912 21:28:12.634446 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2127-11m4e
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230912-2127-11m4e-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0912 21:28:12.705868 140512254185472 k8s_base_runner.py:284] Service psm-grpc-server created
I0912 21:28:12.706134 140512254185472 k8s_base_runner.py:670] Waiting for NEG for service psm-grpc-server
I0912 21:28:12.802839 140512254185472 k8s_base_runner.py:675] Service psm-grpc-server: detected NEG=psm-interop-server-20230912-2127-11m4e-psm-grpc-server in zones=['us-central1-a']
I0912 21:28:12.803056 140512254185472 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2127-11m4e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:28:13.669150 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2127-11m4e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:28:13.729791 140512254185472 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0912 21:28:13.737996 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230912-2127-11m4e
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0ottsfle
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0ottsfle
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0ottsfle
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:558b5b0bfac8e21755c223063274a779b3898afe
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230912-2127-11m4e-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:28:13.806441 140512254185472 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0912 21:28:13.806682 140512254185472 k8s_base_runner.py:605] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0912 21:28:13.962325 140512254185472 k8s_base_runner.py:615] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-5fddfcb799-hrpvw', 'psm-grpc-server-5fddfcb799-mdfzg']
I0912 21:28:13.962697 140512254185472 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-5fddfcb799-hrpvw to start
I0912 21:28:17.202253 140512254185472 k8s_base_runner.py:628] Pod psm-grpc-server-5fddfcb799-hrpvw ready, IP: 10.120.9.147
I0912 21:28:17.202552 140512254185472 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-5fddfcb799-hrpvw to round_robin_test/test_app_logs/psm-interop-server-20230912-2127-11m4e_psm-grpc-server-5fddfcb799-hrpvw.log
I0912 21:28:17.203225 140512163046976 k8s_log_collector.py:70] Starting log collection thread 140512163046976 for psm-grpc-server-5fddfcb799-hrpvw
I0912 21:28:17.203517 140512254185472 k8s_base_runner.py:625] Waiting for pod psm-grpc-server-5fddfcb799-mdfzg to start
I0912 21:28:17.312364 140512254185472 k8s_base_runner.py:628] Pod psm-grpc-server-5fddfcb799-mdfzg ready, IP: 10.120.11.138
I0912 21:28:17.312636 140512254185472 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-server-5fddfcb799-mdfzg to round_robin_test/test_app_logs/psm-interop-server-20230912-2127-11m4e_psm-grpc-server-5fddfcb799-mdfzg.log
I0912 21:28:17.313048 140512154654272 k8s_log_collector.py:70] Starting log collection thread 140512154654272 for psm-grpc-server-5fddfcb799-mdfzg
I0912 21:28:17.313211 140512254185472 k8s_base_runner.py:587] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0912 21:28:23.723306 140512254185472 k8s_base_runner.py:596] Deployment psm-grpc-server has 2 replicas available
I0912 21:28:23.723838 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0912 21:28:23.723967 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0912 21:28:23.760489 140512254185472 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0912 21:28:24.058833 140512254185472 traffic_director.py:255] Loaded NEG "psm-interop-server-20230912-2127-11m4e-psm-grpc-server" in zone us-central1-a
I0912 21:28:24.059062 140512254185472 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230912-2127-11m4e: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230912-2127-11m4e-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2127-11m4e-psm-grpc-server', zone='us-central1-a')}
I0912 21:28:24.063072 140512254185472 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230912-2127-11m4e-psm-grpc-server
maxRatePerEndpoint: 5
...

I0912 21:28:25.173044 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554104065-605301f6cee90-05761d74-74e9be01
I0912 21:28:57.801072 140512254185472 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230912-2127-11m4e to report all backends healthy: ['psm-interop-server-20230912-2127-11m4e-psm-grpc-server']
I0912 21:29:11.256478 140512254185472 compute.py:487] Backend psm-interop-server-20230912-2127-11m4e-psm-grpc-server in zone us-central1-a reported healthy
I0912 21:29:15.261171 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0912 21:29:15.261403 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0912 21:29:15.261536 140512254185472 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230912-2127-11m4e: server_target=xds:///psm-grpc-server:27043 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0912 21:29:15.261772 140512254185472 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2127-11m4e%22;timeRange=2023-09-12T21:29:15.261622Z%2F2023-09-12T21:59:15.261622Z?project=grpc-testing
I0912 21:29:15.263773 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230912-2127-11m4e
labels:
name: psm-interop-client-20230912-2127-11m4e
owner: xds-k8s-interop-test
...

I0912 21:29:15.345638 140512254185472 k8s_base_runner.py:284] Namespace psm-interop-client-20230912-2127-11m4e created
I0912 21:29:15.345884 140512254185472 k8s_base_runner.py:341] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2127-11m4e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:29:16.054916 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2127-11m4e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0912 21:29:16.112156 140512254185472 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0912 21:29:16.120349 140512254185472 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230912-2127-11m4e
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-i4worlrv
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-i4worlrv
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-i4worlrv
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:a02622572aec2b236257b86842d26eb32e5805f0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:27043"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d47f36f78aef8fe9d67303be1aba480f2fe9a77c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0912 21:29:16.193466 140512254185472 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0912 21:29:16.193728 140512254185472 k8s_base_runner.py:605] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0912 21:29:16.300217 140512254185472 k8s_base_runner.py:615] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-84b66675bc-cnzsl']
I0912 21:29:16.300576 140512254185472 k8s_base_runner.py:625] Waiting for pod psm-grpc-client-84b66675bc-cnzsl to start
I0912 21:29:19.531958 140512254185472 k8s_base_runner.py:628] Pod psm-grpc-client-84b66675bc-cnzsl ready, IP: 10.120.10.228
I0912 21:29:19.532235 140512254185472 k8s_base_runner.py:651] Enabling log collection from pod psm-grpc-client-84b66675bc-cnzsl to round_robin_test/test_app_logs/psm-interop-client-20230912-2127-11m4e_psm-grpc-client-84b66675bc-cnzsl.log
I0912 21:29:19.532714 140512146261568 k8s_log_collector.py:70] Starting log collection thread 140512146261568 for psm-grpc-client-84b66675bc-cnzsl
I0912 21:29:19.532858 140512254185472 k8s_base_runner.py:587] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0912 21:29:22.832133 140512254185472 k8s_base_runner.py:596] Deployment psm-grpc-client has 1 replicas available
I0912 21:29:22.832777 140512254185472 client_app.py:200] [psm-grpc-client-84b66675bc-cnzsl] Waiting to report a READY channel to xds:///psm-grpc-server:27043
I0912 21:29:22.958691 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:29:33.046183 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:29:43.128253 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:29:53.211380 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:30:03.296562 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:30:19.386118 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:30:44.484990 140512254185472 client_app.py:236] [psm-grpc-client-84b66675bc-cnzsl] Server channel:
I0912 21:30:44.523499 140512254185472 client_app.py:249] [psm-grpc-client-84b66675bc-cnzsl] Found subchannel in state READY:
I0912 21:30:44.523851 140512254185472 client_app.py:211] [psm-grpc-client-84b66675bc-cnzsl] Channel to xds:///psm-grpc-server:27043 transitioned to state READY:
I0912 21:30:44.524025 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0912 21:30:44.524121 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0912 21:30:44.524496 140512254185472 grpc.py:74] [psm-grpc-client-84b66675bc-cnzsl:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0912 21:30:44.569724 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0912 21:30:44.570052 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0912 21:30:44.570327 140512254185472 grpc.py:74] [psm-grpc-client-84b66675bc-cnzsl:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:30:48.649812 140512254185472 xds_k8s_testcase.py:586] [psm-grpc-client-84b66675bc-cnzsl] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5fddfcb799-mdfzg: 50
psm-grpc-server-5fddfcb799-hrpvw: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-5fddfcb799-mdfzg: 50
psm-grpc-server-5fddfcb799-hrpvw: 50

I0912 21:30:48.650137 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0912 21:30:48.650360 140512254185472 xds_k8s_testcase.py:267] --- Starting subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0912 21:30:48.650546 140512254185472 grpc.py:74] [psm-grpc-client-84b66675bc-cnzsl:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0912 21:30:52.721625 140512254185472 xds_k8s_testcase.py:586] [psm-grpc-client-84b66675bc-cnzsl] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5fddfcb799-mdfzg: 50
psm-grpc-server-5fddfcb799-hrpvw: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-5fddfcb799-mdfzg: 50
psm-grpc-server-5fddfcb799-hrpvw: 50

I0912 21:30:52.722064 140512254185472 xds_k8s_testcase.py:272] --- Finished subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0912 21:30:52.722263 140512254185472 xds_k8s_testcase.py:672] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0912 21:30:52.834817 140512254185472 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230912-2127-11m4e"
I0912 21:30:53.640219 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554252839-60530284b09ef-972bdf1c-438612d0
I0912 21:31:15.515452 140512254185472 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230912-2127-11m4e"
I0912 21:31:15.650810 140512254185472 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:15.651054 140512254185472 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230912-2127-11m4e"
I0912 21:31:16.104628 140512254185472 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:16.104902 140512254185472 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230912-2127-11m4e"
I0912 21:31:16.776652 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554276108-6053029ae17fa-14e62e06-f41683f7
I0912 21:31:19.135228 140512254185472 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230912-2127-11m4e"
I0912 21:31:19.485831 140512254185472 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:19.486066 140512254185472 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230912-2127-11m4e"
I0912 21:31:20.128318 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554279491-6053029e1b984-3a305ca5-c395823e
I0912 21:31:22.512090 140512254185472 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230912-2127-11m4e"
I0912 21:31:23.002436 140512254185472 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:23.002695 140512254185472 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230912-2127-11m4e"
I0912 21:31:23.828945 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554283008-605302a1760d0-f93ae3ad-743f0042
I0912 21:31:52.296020 140512254185472 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230912-2127-11m4e"
I0912 21:31:52.712886 140512254185472 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:52.713145 140512254185472 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230912-2127-11m4e"
I0912 21:31:53.175933 140512254185472 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230912-2127-11m4e" not deleted since it does not exist
I0912 21:31:53.176165 140512254185472 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230912-2127-11m4e"
I0912 21:31:53.730511 140512254185472 compute.py:589] Waiting 600 sec for compute operation id: operation-1694554313267-605302be5175b-e3ea833e-8d3de095
I0912 21:31:58.197176 140512254185472 k8s_base_runner.py:535] Deleting deployment psm-grpc-client
I0912 21:32:38.627527 140512254185472 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230912-2127-11m4e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:32:39.209355 140512254185472 k8s_base_runner.py:561] Deleting service account psm-grpc-client
I0912 21:32:39.340759 140512254185472 k8s_base_runner.py:573] Deleting namespace psm-interop-client-20230912-2127-11m4e
I0912 21:33:09.507530 140512254185472 k8s_base_runner.py:535] Deleting deployment psm-grpc-server
I0912 21:33:19.685502 140512254185472 k8s_base_runner.py:548] Deleting service psm-grpc-server
I0912 21:33:19.800262 140512254185472 k8s_base_runner.py:360] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230912-2127-11m4e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0912 21:33:20.491464 140512254185472 k8s_base_runner.py:561] Deleting service account psm-grpc-server
I0912 21:33:20.601920 140512254185472 k8s_base_runner.py:573] Deleting namespace psm-interop-server-20230912-2127-11m4e
I0912 21:33:50.787530 140512254185472 xds_k8s_testcase.py:696] ----- Test client/server logs -----
I0912 21:33:50.787985 140512254185472 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-client-i4worlrv:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230912-2127-11m4e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-i4worlrv%22;timeRange=2023-09-12T21:29:15.261614Z%2F2023-09-12T21:33:09.507481Z?project=grpc-testing
I0912 21:33:50.788187 140512254185472 k8s_base_runner.py:746] GCP Logs Explorer link to psm-grpc-server-0ottsfle:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230912-2127-11m4e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-0ottsfle%22;timeRange=2023-09-12T21:28:12.491412Z%2F2023-09-12T21:33:50.787367Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
I0912 21:33:50.788497 140512254185472 base_testcase.py:52] ----- TestCase __main__.RoundRobinTest.test_round_robin PASSED -----
----------------------------------------------------------------------
Ran 1 test in 380.721s

OK
Failed test suites: 0


[ID: 7548055] Command finished after 5501 secs, exit value: 0


Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[14:33:52] Collecting build artifacts from build VM
[14:34:04] Kokoro builder finished