Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[16:19:41 PDT] Transferring environment variable script to build VM
[16:19:41 PDT] Transferring kokoro_log_reader.py to build VM
[16:19:41 PDT] Transferring source code to build VM
[16:19:42 PDT] Executing build script on build VM



[ID: 4869297] Executing command via SSH:
export KOKORO_BUILD_NUMBER="719"
export KOKORO_JOB_NAME="grpc/node/master/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

Submodule 'packages/grpc-js-xds/deps/envoy-api' (https://github.com/envoyproxy/data-plane-api.git) registered for path '../deps/envoy-api'
Submodule 'packages/grpc-js-xds/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../deps/googleapis'
Submodule 'packages/grpc-js-xds/deps/protoc-gen-validate' (https://github.com/envoyproxy/protoc-gen-validate.git) registered for path '../deps/protoc-gen-validate'
Submodule 'packages/grpc-js-xds/deps/xds' (https://github.com/cncf/xds.git) registered for path '../deps/xds'
Submodule 'packages/grpc-tools/deps/protobuf' (https://github.com/protocolbuffers/protobuf) registered for path '../../grpc-tools/deps/protobuf'
Submodule 'packages/proto-loader/deps/gapic-showcase' (https://github.com/googleapis/gapic-showcase.git) registered for path '../../proto-loader/deps/gapic-showcase'
Submodule 'packages/proto-loader/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../../proto-loader/deps/googleapis'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/envoy-api'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/googleapis'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/protoc-gen-validate'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/xds'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/googleapis'...
Submodule path '../deps/envoy-api': checked out '78f198cf96ecdc7120ef640406770aa01af775c4'
Submodule path '../deps/googleapis': checked out '8c53b2cb792234354c13336ac7daee61333deade'
Submodule path '../deps/protoc-gen-validate': checked out '0af61d9dc28712dc0e6f8e1a940855a2ee0cb9ed'
Submodule path '../deps/xds': checked out 'cb28da3451f158a947dfc45090fe92b07b243bc1'
Submodule path '../../grpc-tools/deps/protobuf': checked out '7c40b2df1fdf6f414c1c18c789715a9c948a0725'
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path '../../grpc-tools/deps/protobuf/third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path '../../grpc-tools/deps/protobuf/third_party/googletest'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/benchmark'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/googletest'...
Submodule path '../../grpc-tools/deps/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
Submodule path '../../grpc-tools/deps/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
Submodule path '../../proto-loader/deps/gapic-showcase': checked out 'b09b3ba9a8db8aae7d5d7c3939853681cc97c293'
Submodule 'schema/api-common-protos' (https://github.com/googleapis/api-common-protos.git) registered for path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos'
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase/schema/api-common-protos'...
Submodule path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos': checked out '4c0a203e3658ae0e56d47c817c2c5904116c0ae0'
Submodule path '../../proto-loader/deps/googleapis': checked out '8f2eda119e11c8bd0c189b545da18bba9019c83e'
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: SECONDARY_GKE_CLUSTER_NAME=psm-interop-lb-secondary SECONDARY_GKE_CLUSTER_ZONE=us-west1-b
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-node
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/sergiitk/grpc-node.git
GIT_COMMIT_SHORT,ade39e8b
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 135361 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 135306 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
Selecting previously unselected package google-cloud-cli-gke-gcloud-auth-plugin.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 134906 files and directories currently installed.)
Preparing to unpack .../0-google-cloud-cli-gke-gcloud-auth-plugin_471.0.0-0_amd64.deb ...
Unpacking google-cloud-cli-gke-gcloud-auth-plugin (471.0.0-0) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
Preparing to unpack .../1-google-cloud-sdk-gke-gcloud-auth-plugin_467.0.0-0_all.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (467.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../2-kubectl_1%3a471.0.0-0_amd64.deb ...
Unpacking kubectl (1:471.0.0-0) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../3-python3-pip-whl_22.0.2+dfsg-1ubuntu0.4_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../4-python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../5-python3.10-venv_3.10.12-1~22.04.3_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.3) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Setting up google-cloud-cli-gke-gcloud-auth-plugin (471.0.0-0) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (467.0.0-0) ...
Setting up kubectl (1:471.0.0-0) ...
Setting up python3.10-venv (3.10.12-1~22.04.3) ...
Processing triggers for man-db (2.10.2-1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.S3tOHHXyEN/psm-interop
Cloning into '/tmp/tmp.S3tOHHXyEN/psm-interop'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.S3tOHHXyEN/psm-interop/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl.metadata (2.9 kB)
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl.metadata (2.0 kB)
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl.metadata (2.4 kB)
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl.metadata (5.1 kB)
Collecting google-cloud-monitoring==2.18.0 (from -r requirements.lock (line 6))
Downloading google_cloud_monitoring-2.18.0-py2.py3-none-any.whl.metadata (5.6 kB)
Collecting grpcio==1.60.1 (from -r requirements.lock (line 7))
Downloading grpcio-1.60.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting grpcio-health-checking==1.60.1 (from -r requirements.lock (line 8))
Downloading grpcio_health_checking-1.60.1-py3-none-any.whl.metadata (1.3 kB)
Collecting grpcio-tools==1.60.1 (from -r requirements.lock (line 9))
Downloading grpcio_tools-1.60.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio-channelz==1.60.1 (from -r requirements.lock (line 10))
Downloading grpcio_channelz-1.60.1-py3-none-any.whl.metadata (1.3 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 11))
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 12))
Downloading six-1.16.0-py2.py3-none-any.whl.metadata (1.8 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 13))
Downloading tenacity-6.3.1-py2.py3-none-any.whl.metadata (1.2 kB)
Collecting typing_extensions==4.10.0 (from -r requirements.lock (line 14))
Downloading typing_extensions-4.10.0-py3-none-any.whl.metadata (3.0 kB)
Collecting packaging==23.1 (from -r requirements.lock (line 15))
Downloading packaging-23.1-py3-none-any.whl.metadata (3.1 kB)
Collecting Pygments==2.14.0 (from -r requirements.lock (line 16))
Downloading Pygments-2.14.0-py3-none-any.whl.metadata (1.6 kB)
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 17))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl.metadata (8.2 kB)
Collecting protobuf==4.24.1 (from -r requirements.lock (line 18))
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting xds-protos==1.58.0rc1 (from -r requirements.lock (line 19))
Downloading xds_protos-1.58.0rc1-py3-none-any.whl.metadata (1.3 kB)
Collecting cachetools==5.3.1 (from -r requirements.lock (line 21))
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.lock (line 22))
Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.2.0 (from -r requirements.lock (line 23))
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting google-api-core==2.11.1 (from -r requirements.lock (line 24))
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0 (from -r requirements.lock (line 25))
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 26))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl.metadata (2.1 kB)
Collecting googleapis-common-protos==1.60.0 (from -r requirements.lock (line 27))
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 28))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl.metadata (3.2 kB)
Collecting grpcio-status==1.60.1 (from -r requirements.lock (line 29))
Downloading grpcio_status-1.60.1-py3-none-any.whl.metadata (1.3 kB)
Collecting httplib2==0.22.0 (from -r requirements.lock (line 30))
Downloading httplib2-0.22.0-py3-none-any.whl.metadata (2.6 kB)
Collecting idna==3.4 (from -r requirements.lock (line 31))
Downloading idna-3.4-py3-none-any.whl.metadata (9.8 kB)
Collecting MarkupSafe==2.1.3 (from -r requirements.lock (line 32))
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 33))
Downloading oauthlib-3.2.2-py3-none-any.whl.metadata (7.5 kB)
Collecting proto-plus==1.22.3 (from -r requirements.lock (line 34))
Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.lock (line 35))
Downloading pyasn1-0.5.0-py2.py3-none-any.whl.metadata (8.5 kB)
Collecting pyasn1-modules==0.3.0 (from -r requirements.lock (line 36))
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl.metadata (3.6 kB)
Collecting pyparsing==3.1.1 (from -r requirements.lock (line 37))
Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting requests==2.31.0 (from -r requirements.lock (line 38))
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 39))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl.metadata (10 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 40))
Downloading rsa-4.9-py3-none-any.whl.metadata (4.2 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 41))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl.metadata (4.6 kB)
Collecting urllib3==1.26.16 (from -r requirements.lock (line 42))
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 2.5 MB/s eta 0:00:00
Collecting websocket-client==1.6.2 (from -r requirements.lock (line 43))
Downloading websocket_client-1.6.2-py3-none-any.whl.metadata (7.5 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.60.1->-r requirements.lock (line 9)) (69.2.0)
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 3.7 MB/s eta 0:00:00
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 7.2 MB/s eta 0:00:00
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 16.4 MB/s eta 0:00:00
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 8.4 MB/s eta 0:00:00
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 13.7 MB/s eta 0:00:00
Downloading google_cloud_monitoring-2.18.0-py2.py3-none-any.whl (325 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 326.0/326.0 kB 25.0 MB/s eta 0:00:00
Downloading grpcio-1.60.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.4/5.4 MB 44.6 MB/s eta 0:00:00
Downloading grpcio_health_checking-1.60.1-py3-none-any.whl (18 kB)
Downloading grpcio_tools-1.60.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.8/2.8 MB 95.1 MB/s eta 0:00:00
Downloading grpcio_channelz-1.60.1-py3-none-any.whl (23 kB)
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 77.7 MB/s eta 0:00:00
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Downloading typing_extensions-4.10.0-py3-none-any.whl (33 kB)
Downloading packaging-23.1-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 6.4 MB/s eta 0:00:00
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 65.4 MB/s eta 0:00:00
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 25.5 MB/s eta 0:00:00
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.4/311.4 kB 34.2 MB/s eta 0:00:00
Downloading xds_protos-1.58.0rc1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 48.5 MB/s eta 0:00:00
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 21.9 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.8/201.8 kB 27.3 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 16.0 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 24.0 MB/s eta 0:00:00
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 28.4 MB/s eta 0:00:00
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Downloading grpcio_status-1.60.1-py3-none-any.whl (14 kB)
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 10.2 MB/s eta 0:00:00
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 8.1 MB/s eta 0:00:00
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 21.4 MB/s eta 0:00:00
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 6.6 MB/s eta 0:00:00
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 11.4 MB/s eta 0:00:00
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 23.7 MB/s eta 0:00:00
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 14.2 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 9.0 MB/s eta 0:00:00
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 18.3 MB/s eta 0:00:00
Downloading websocket_client-1.6.2-py3-none-any.whl (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 7.4 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, uritemplate, typing_extensions, six, PyYAML, pyparsing, Pygments, pyasn1, protobuf, packaging, oauthlib, MarkupSafe, idna, grpcio, charset-normalizer, certifi, cachetools, xds-protos, tenacity, rsa, requests, python-dateutil, pyasn1-modules, proto-plus, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, absl-py, requests-oauthlib, grpcio-status, google-auth, kubernetes, grpc-google-iam-v1, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-secret-manager, google-cloud-monitoring
Successfully installed Mako-1.2.4 MarkupSafe-2.1.3 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-api-core-2.11.1 google-api-python-client-1.12.11 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-cloud-monitoring-2.18.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.60.1 grpcio-channelz-1.60.1 grpcio-health-checking-1.60.1 grpcio-status-1.60.1 grpcio-tools-1.60.1 httplib2-0.22.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.1 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 typing_extensions-4.10.0 uritemplate-3.0.1 urllib3-1.26.16 websocket-client-1.6.2 xds-protos-1.58.0rc1
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.1
certifi 2023.7.22
charset-normalizer 3.2.0
google-api-core 2.11.1
google-api-python-client 1.12.11
google-auth 2.22.0
google-auth-httplib2 0.1.0
google-cloud-monitoring 2.18.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.60.0
grpc-google-iam-v1 0.12.6
grpcio 1.60.1
grpcio-channelz 1.60.1
grpcio-health-checking 1.60.1
grpcio-status 1.60.1
grpcio-tools 1.60.1
httplib2 0.22.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.3
oauthlib 3.2.2
packaging 23.1
pip 24.0
proto-plus 1.22.3
protobuf 4.24.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
Pygments 2.14.0
pyparsing 3.1.1
python-dateutil 2.8.2
PyYAML 6.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.2.0
six 1.16.0
tenacity 6.3.1
typing_extensions 4.10.0
uritemplate 3.0.1
urllib3 1.26.16
websocket-client 1.6.2
xds-protos 1.58.0rc1
Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto
/tmp/tmp.S3tOHHXyEN/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.S3tOHHXyEN/psm-interop/protos/grpc/testing:
total 108
-rw-rw-r-- 1 kbuilder kbuilder 957 Apr 8 23:20 empty.proto
-rw-rw-r-- 1 kbuilder kbuilder 1040 Apr 8 23:20 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Apr 8 23:20 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 11527 Apr 8 23:20 messages.proto
-rw-rw-r-- 1 kbuilder kbuilder 17599 Apr 8 23:20 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Apr 8 23:20 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 4871 Apr 8 23:20 test.proto
-rw-rw-r-- 1 kbuilder kbuilder 4353 Apr 8 23:20 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 41870 Apr 8 23:20 test_pb2_grpc.py
Client image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
Client image not found
Building Node xDS interop test app Docker images
/tmpfs/src/github/grpc-node /tmp/tmp.S3tOHHXyEN/psm-interop
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.87kB done
#1 DONE 0.0s

#2 [internal] load metadata for docker.io/library/node:18-slim
#2 ...

#3 [internal] load metadata for gcr.io/distroless/nodejs18-debian11:latest
#3 DONE 0.6s

#2 [internal] load metadata for docker.io/library/node:18-slim
#2 DONE 0.7s

#4 [internal] load .dockerignore
#4 transferring context: 2B done
#4 DONE 0.0s

#5 [build 1/9] FROM docker.io/library/node:18-slim@sha256:c569d19289293797aefad0473a046d882f1b9eb0f444e9a0018648d5fd03c812
#5 resolve docker.io/library/node:18-slim@sha256:c569d19289293797aefad0473a046d882f1b9eb0f444e9a0018648d5fd03c812 0.0s done
#5 sha256:503fbb4f74dfc223fa7df9ea3a583f85ae96f7c16c55b77d692ffa77d56a8b30 0B / 3.35kB 0.1s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 0B / 38.16MB 0.1s
#5 sha256:c569d19289293797aefad0473a046d882f1b9eb0f444e9a0018648d5fd03c812 1.21kB / 1.21kB done
#5 sha256:ac5fac99b7e68b660bbd22e847b4a295be402da2f6b756865ca14be901a63e01 1.37kB / 1.37kB done
#5 sha256:e166efd45c79a5aec2e53f567d143e51956f9489d49ad710e1bfc0a50671b89d 7.67kB / 7.67kB done
#5 sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 0B / 29.12MB 0.1s
#5 sha256:503fbb4f74dfc223fa7df9ea3a583f85ae96f7c16c55b77d692ffa77d56a8b30 3.35kB / 3.35kB 0.1s done
#5 sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 9.38MB / 29.12MB 0.2s
#5 sha256:737c1dca3024689406e893660d54a2c3a6627a6adf82452b4baac73fdc492dfc 0B / 2.67MB 0.2s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 3.15MB / 38.16MB 0.3s
#5 sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 16.78MB / 29.12MB 0.3s
#5 sha256:737c1dca3024689406e893660d54a2c3a6627a6adf82452b4baac73fdc492dfc 2.10MB / 2.67MB 0.3s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 7.34MB / 38.16MB 0.4s
#5 sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 25.17MB / 29.12MB 0.4s
#5 sha256:737c1dca3024689406e893660d54a2c3a6627a6adf82452b4baac73fdc492dfc 2.67MB / 2.67MB 0.3s done
#5 sha256:26417687e75d50777d9ade53f5c575c72e567dc8b341671eaeb425a7630af9ca 450B / 450B 0.4s done
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 16.78MB / 38.16MB 0.5s
#5 sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 29.12MB / 29.12MB 0.5s done
#5 extracting sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 29.36MB / 38.16MB 0.7s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 31.46MB / 38.16MB 0.8s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 36.70MB / 38.16MB 0.9s
#5 sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 38.16MB / 38.16MB 1.0s done
#5 extracting sha256:8a1e25ce7c4f75e372e9884f8f7b1bedcfe4a7a7d452eb4b0a1c7477c9a90345 2.2s done
#5 extracting sha256:503fbb4f74dfc223fa7df9ea3a583f85ae96f7c16c55b77d692ffa77d56a8b30
#5 extracting sha256:503fbb4f74dfc223fa7df9ea3a583f85ae96f7c16c55b77d692ffa77d56a8b30 done
#5 extracting sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09
#5 ...

#6 [internal] load build context
#6 transferring context: 736.61MB 4.0s done
#6 DONE 4.1s

#5 [build 1/9] FROM docker.io/library/node:18-slim@sha256:c569d19289293797aefad0473a046d882f1b9eb0f444e9a0018648d5fd03c812
#5 ...

#7 [stage-1 1/5] FROM gcr.io/distroless/nodejs18-debian11:latest@sha256:1aedd34f20178c18d212a06c059b956a64b60ef74325f07c8066c6107b6770fb
#7 resolve gcr.io/distroless/nodejs18-debian11:latest@sha256:1aedd34f20178c18d212a06c059b956a64b60ef74325f07c8066c6107b6770fb 0.0s done
#7 sha256:d5cab0f497557bb42a3dd73b20e238a6ba3e3d64de6097a4e2fe9b1b169c9728 2.28kB / 2.28kB done
#7 sha256:53e59478744302c82612c3683b900716c6a9fbcc5660dd52315ed17e04ddd0af 3.05kB / 3.05kB done
#7 sha256:1aedd34f20178c18d212a06c059b956a64b60ef74325f07c8066c6107b6770fb 1.51kB / 1.51kB done
#7 sha256:16100daf795a61c7588657ad2b6afc6eef7300de0c874b6c3d4def1796e28081 103.77kB / 103.77kB 0.5s done
#7 extracting sha256:16100daf795a61c7588657ad2b6afc6eef7300de0c874b6c3d4def1796e28081 done
#7 sha256:e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4 21.19kB / 21.19kB 0.5s done
#7 sha256:473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e 755.28kB / 755.28kB 0.6s done
#7 extracting sha256:e33bce57de289fffd2380f73997dfb7e1ec193877904bed99f28c715d071fdc4 done
#7 extracting sha256:473d8557b1b27974f7dc7c4b4e1a209df0e27e8cae1e3e33b7bb45c969b6fc7e 0.6s done
#7 sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71 67B / 67B 0.6s done
#7 sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd 168B / 168B 0.7s done
#7 sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3 93B / 93B 0.7s done
#7 sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 188B / 188B 0.6s done
#7 sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8 122B / 122B 0.7s done
#7 sha256:9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317 320B / 320B 0.8s done
#7 sha256:9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0 122.56kB / 122.56kB 0.9s done
#7 sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f 385B / 385B 0.8s done
#7 sha256:59e6e6563fee04c9590afd06f4a732b9af37f4804248e5ee49305b6e268de8e5 5.85MB / 5.85MB 1.1s done
#7 sha256:a4ba90834fb4abf3d80bbdaaaef36560ab1bb682f5279d44114d768e119639b9 2.06MB / 2.06MB 1.0s done
#7 sha256:df368711b36276ed02b2040d3e3296b919042d2a05a2bbe9f758e708436c12cf 968.57kB / 968.57kB 1.1s done
#7 sha256:e89169bec965f3b73bc8a6cf650d362a1da8ae571c5fdccccf7bab8194c03c47 131.15kB / 131.15kB 1.1s done
#7 sha256:7f3501c931c2f262add9c59a46b7469c09f862cfcdb005d9187bc4f8f9964724 741.33kB / 741.33kB 1.2s done
#7 sha256:1018ee1ac94a4d3b0ed0de651532906f54c6eb3bf625f93d5867f3e191edd451 50.15MB / 50.15MB 1.6s done
#7 sha256:88934a1bc18c35762f0009512e7a775aec9e1d9531a1781a0caf81013a67ab94 52.41kB / 52.41kB 1.2s done
#7 extracting sha256:b6824ed73363f94b3b2b44084c51c31bc32af77a96861d49e16f91e3ab6bed71 done
#7 extracting sha256:7c12895b777bcaa8ccae0605b4de635b68fc32d60fa08f421dc3818bf55ee212 done
#7 extracting sha256:33e068de264953dfdc9f9ada207e76b61159721fd64a4820b320d05133a55fb8 done
#7 extracting sha256:5664b15f108bf9436ce3312090a767300800edbbfd4511aa1a6d64357024d5dd done
#7 extracting sha256:27be814a09ebd97fac6fb7b82d19f117185e90601009df3fbab6f442f85cd6b3 done
#7 extracting sha256:4aa0ea1413d37a58615488592a0b827ea4b2e48fa5a77cf707d0e35f025e613f done
#7 extracting sha256:9ef7d74bdfdf3c517b28bd694a9159e94e5f53ff1ca87b39f8ca1ac0be2ed317 done
#7 extracting sha256:9112d77ee5b16873acaa186b816c3c61f5f8eba40730e729e9614a27f40211e0 done
#7 extracting sha256:59e6e6563fee04c9590afd06f4a732b9af37f4804248e5ee49305b6e268de8e5 0.2s done
#7 extracting sha256:a4ba90834fb4abf3d80bbdaaaef36560ab1bb682f5279d44114d768e119639b9 0.0s done
#7 extracting sha256:df368711b36276ed02b2040d3e3296b919042d2a05a2bbe9f758e708436c12cf 0.0s done
#7 extracting sha256:e89169bec965f3b73bc8a6cf650d362a1da8ae571c5fdccccf7bab8194c03c47 done
#7 extracting sha256:7f3501c931c2f262add9c59a46b7469c09f862cfcdb005d9187bc4f8f9964724 0.0s done
#7 extracting sha256:88934a1bc18c35762f0009512e7a775aec9e1d9531a1781a0caf81013a67ab94 done
#7 extracting sha256:1018ee1ac94a4d3b0ed0de651532906f54c6eb3bf625f93d5867f3e191edd451 2.0s done
#7 DONE 4.4s

#5 [build 1/9] FROM docker.io/library/node:18-slim@sha256:c569d19289293797aefad0473a046d882f1b9eb0f444e9a0018648d5fd03c812
#5 extracting sha256:f22ebab8fc9f8fcaa6df64bc292636e991d7926a18a000daa60df7ec7a508d09 1.6s done
#5 extracting sha256:737c1dca3024689406e893660d54a2c3a6627a6adf82452b4baac73fdc492dfc
#5 extracting sha256:737c1dca3024689406e893660d54a2c3a6627a6adf82452b4baac73fdc492dfc 0.3s done
#5 extracting sha256:26417687e75d50777d9ade53f5c575c72e567dc8b341671eaeb425a7630af9ca
#5 extracting sha256:26417687e75d50777d9ade53f5c575c72e567dc8b341671eaeb425a7630af9ca done
#5 DONE 6.8s

#8 [stage-1 2/5] WORKDIR /node/src/grpc-node
#8 ...

#9 [build 2/9] WORKDIR /node/src/grpc-node
#9 DONE 0.9s

#8 [stage-1 2/5] WORKDIR /node/src/grpc-node
#8 DONE 3.2s

#10 [build 3/9] COPY . .
#10 DONE 4.6s

#11 [build 4/9] WORKDIR /node/src/grpc-node/packages/proto-loader
#11 DONE 0.0s

#12 [build 5/9] RUN npm install
#12 14.92
#12 14.92 > @grpc/proto-loader@0.7.12 prepare
#12 14.92 > npm run compile
#12 14.92
#12 15.24
#12 15.24 > @grpc/proto-loader@0.7.12 compile
#12 15.24 > tsc -p .
#12 15.24
#12 17.26
#12 17.26 added 300 packages, and audited 301 packages in 17s
#12 17.26
#12 17.26 49 packages are looking for funding
#12 17.26 run `npm fund` for details
#12 17.27
#12 17.27 found 0 vulnerabilities
#12 17.27 npm notice
#12 17.27 npm notice New patch version of npm available! 10.5.0 -> 10.5.1
#12 17.27 npm notice Changelog:
#12 17.27 npm notice Run `npm install -g npm@10.5.1` to update!
#12 17.27 npm notice
#12 DONE 18.1s

#13 [build 6/9] WORKDIR /node/src/grpc-node/packages/grpc-js
#13 DONE 0.0s

#14 [build 7/9] RUN npm install
#14 12.91 npm WARN deprecated flatten@1.0.3: flatten is deprecated in favor of utility frameworks such as lodash.
#14 12.96 npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
#14 12.98 npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
#14 12.98 npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
#14 13.05 npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
#14 15.00 npm WARN deprecated chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
#14 15.07 npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
#14 19.41
#14 19.41 > @grpc/grpc-js@1.9.14 prepare
#14 19.41 > npm run generate-types && npm run compile
#14 19.41
#14 19.71
#14 19.71 > @grpc/grpc-js@1.9.14 generate-types
#14 19.71 > proto-loader-gen-types --keepCase --longs String --enums String --defaults --oneofs --includeComments --includeDirs proto/ --include-dirs test/fixtures/ -O src/generated/ --grpcLib ../index channelz.proto
#14 19.71
#14 20.16
#14 20.16 > @grpc/grpc-js@1.9.14 compile
#14 20.16 > tsc -p .
#14 20.16
#14 24.75
#14 24.75 added 706 packages, and audited 708 packages in 24s
#14 24.75
#14 24.75 81 packages are looking for funding
#14 24.75 run `npm fund` for details
#14 24.76
#14 24.76 11 vulnerabilities (1 moderate, 9 high, 1 critical)
#14 24.76
#14 24.76 To address issues that do not require attention, run:
#14 24.76 npm audit fix
#14 24.76
#14 24.76 To address all issues (including breaking changes), run:
#14 24.76 npm audit fix --force
#14 24.76
#14 24.76 Run `npm audit` for details.
#14 DONE 25.9s

#15 [build 8/9] WORKDIR /node/src/grpc-node/packages/grpc-js-xds
#15 DONE 0.0s

#16 [build 9/9] RUN npm install
#16 12.90
#16 12.90 > @grpc/grpc-js-xds@1.9.2 prepare
#16 12.90 > npm run generate-types && npm run compile
#16 12.90
#16 13.20
#16 13.20 > @grpc/grpc-js-xds@1.9.2 generate-types
#16 13.20 > proto-loader-gen-types --keepCase --longs String --enums String --defaults --oneofs --includeComments --includeDirs deps/envoy-api/ deps/xds/ deps/googleapis/ deps/protoc-gen-validate/ -O src/generated/ --grpcLib @grpc/grpc-js envoy/service/discovery/v3/ads.proto envoy/service/load_stats/v3/lrs.proto envoy/config/listener/v3/listener.proto envoy/config/route/v3/route.proto envoy/config/cluster/v3/cluster.proto envoy/config/endpoint/v3/endpoint.proto envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto udpa/type/v1/typed_struct.proto xds/type/v3/typed_struct.proto envoy/extensions/filters/http/fault/v3/fault.proto envoy/service/status/v3/csds.proto envoy/extensions/load_balancing_policies/wrr_locality/v3/wrr_locality.proto envoy/extensions/load_balancing_policies/ring_hash/v3/ring_hash.proto envoy/extensions/load_balancing_policies/pick_first/v3/pick_first.proto
#16 13.20
#16 14.46
#16 14.46 > @grpc/grpc-js-xds@1.9.2 compile
#16 14.46 > tsc
#16 14.46
#16 18.54
#16 18.54 added 316 packages, and audited 319 packages in 18s
#16 18.54
#16 18.54 63 packages are looking for funding
#16 18.54 run `npm fund` for details
#16 18.54
#16 18.54 found 0 vulnerabilities
#16 DONE 18.9s

#17 [stage-1 3/5] COPY --from=build /node/src/grpc-node/packages/proto-loader ./packages/proto-loader/
#17 DONE 2.6s

#18 [stage-1 4/5] COPY --from=build /node/src/grpc-node/packages/grpc-js ./packages/grpc-js/
#18 DONE 2.8s

#19 [stage-1 5/5] COPY --from=build /node/src/grpc-node/packages/grpc-js-xds ./packages/grpc-js-xds/
#19 DONE 2.7s

#20 exporting to image
#20 exporting layers
#20 exporting layers 1.4s done
#20 writing image sha256:9fe9d910fa4b6e9b8eca5d014bbe7ad98270892c85f4dcc90641bc97e741558c
#20 writing image sha256:9fe9d910fa4b6e9b8eca5d014bbe7ad98270892c85f4dcc90641bc97e741558c done
#20 naming to us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0 done
#20 DONE 1.5s
Adding credentials for: us-docker.pkg.dev
Docker configuration file updated.
The push refers to repository [us-docker.pkg.dev/grpc-testing/psm-interop/node-client]
933dfcf521aa: Preparing
b872f41c61d5: Preparing
a7186f45e797: Preparing
4ec7f7f1d0af: Preparing
b1e58a62e761: Preparing
f3aaf0cdb0a8: Preparing
6ea409d80cd8: Preparing
f63d186f6e7d: Preparing
6835249f577a: Preparing
f3aaf0cdb0a8: Waiting
f63d186f6e7d: Waiting
6835249f577a: Waiting
6ea409d80cd8: Waiting
24aacbf97031: Preparing
570759f468b1: Preparing
2388d21e8e2b: Preparing
c048279a7d9f: Preparing
1a73b54f556b: Preparing
2a92d6ac9e4f: Preparing
bbb6cacb8c82: Preparing
ac805962e479: Preparing
af5aa97ebe6c: Preparing
4d049f83d9cf: Preparing
9ed498e122b2: Preparing
577c8ee06f39: Preparing
32ae37dc07be: Preparing
2388d21e8e2b: Waiting
c048279a7d9f: Waiting
1a73b54f556b: Waiting
2a92d6ac9e4f: Waiting
bbb6cacb8c82: Waiting
ac805962e479: Waiting
af5aa97ebe6c: Waiting
4d049f83d9cf: Waiting
9ed498e122b2: Waiting
577c8ee06f39: Waiting
32ae37dc07be: Waiting
24aacbf97031: Waiting
570759f468b1: Waiting
b1e58a62e761: Layer already exists
f3aaf0cdb0a8: Layer already exists
6ea409d80cd8: Layer already exists
f63d186f6e7d: Layer already exists
4ec7f7f1d0af: Pushed
6835249f577a: Layer already exists
24aacbf97031: Layer already exists
2388d21e8e2b: Layer already exists
570759f468b1: Layer already exists
1a73b54f556b: Layer already exists
c048279a7d9f: Layer already exists
2a92d6ac9e4f: Layer already exists
bbb6cacb8c82: Layer already exists
ac805962e479: Layer already exists
af5aa97ebe6c: Layer already exists
4d049f83d9cf: Layer already exists
9ed498e122b2: Layer already exists
577c8ee06f39: Layer already exists
32ae37dc07be: Layer already exists
933dfcf521aa: Pushed
a7186f45e797: Pushed
b872f41c61d5: Pushed
ade39e8baa9c327626f83b862195dbb79ea722a0: digest: sha256:fa9222cef74734f114d81c1d52912bd97df655c4d7d5dbce4e3dc93c9275b335 size: 4919
/tmp/tmp.S3tOHHXyEN/psm-interop
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/affinity_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0408 23:22:46.032605 125727502180352 xds_k8s_testcase.py:162] ----- Testing AffinityTest -----
I0408 23:22:46.032716 125727502180352 xds_k8s_testcase.py:163] Logs timezone: UTC
I0408 23:22:46.032910 125727502180352 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0408 23:22:46.045163 125727502180352 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0408 23:22:46.087491 125727502180352 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0408 23:22:46.271669 125727502180352 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240408-2322-8ocqv
I0408 23:22:46.604566 125727502180352 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/affinity_test
I0408 23:22:46.943043 125727502180352 xds_k8s_testcase.py:731] Found unused xds port: 54704
I0408 23:22:46.943339 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.00_create_health_check ---
I0408 23:22:46.943471 125727502180352 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240408-2322-8ocqv"
I0408 23:22:46.945801 125727502180352 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240408-2322-8ocqv
type: GRPC
...

I0408 23:22:47.700604 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618566997-6159e1552557c-c80502b0-d734f05b
I0408 23:22:49.994355 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.00_create_health_check ---
I0408 23:22:49.994567 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.01_create_backend_services ---
I0408 23:22:49.994680 125727502180352 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240408-2322-8ocqv"
I0408 23:22:50.000735 125727502180352 compute.py:537] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2322-8ocqv
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20240408-2322-8ocqv
protocol: GRPC
sessionAffinity: HEADER_FIELD
...

I0408 23:22:50.790078 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618570002-6159e15803150-e856be67-7822303f
I0408 23:23:18.932341 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.01_create_backend_services ---
I0408 23:23:18.932531 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.02_create_url_map ---
I0408 23:23:18.932614 125727502180352 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240408-2322-8ocqv": psm-grpc-server:54704 -> psm-interop-backend-service-20240408-2322-8ocqv
I0408 23:23:18.939781 125727502180352 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2322-8ocqv
hostRules:
- hosts:
- psm-grpc-server:54704
pathMatcher: psm-interop-path-matcher-20240408-2322-8ocqv
name: psm-interop-url-map-20240408-2322-8ocqv
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2322-8ocqv
name: psm-interop-path-matcher-20240408-2322-8ocqv
...

I0408 23:23:19.695105 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618598941-6159e1739c4c4-a66313cf-caf603d8
I0408 23:23:21.985378 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.02_create_url_map ---
I0408 23:23:21.985581 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.03_create_target_proxy ---
I0408 23:23:21.985688 125727502180352 traffic_director.py:578] Creating target psm-interop-target-proxy-20240408-2322-8ocqv proxy "GRPC" to URL map psm-interop-url-map-20240408-2322-8ocqv
I0408 23:23:21.987328 125727502180352 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240408-2322-8ocqv
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240408-2322-8ocqv
validate_for_proxyless: true
...

I0408 23:23:22.644322 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618601989-6159e17684725-83be264c-f124fd39
I0408 23:23:24.926813 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.03_create_target_proxy ---
I0408 23:23:24.927067 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.04_create_forwarding_rule ---
I0408 23:23:24.927202 125727502180352 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240408-2322-8ocqv" in network "default-vpc": 0.0.0.0:54704 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2322-8ocqv
I0408 23:23:24.929107 125727502180352 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240408-2322-8ocqv
network: global/networks/default-vpc
portRange: 54704
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2322-8ocqv
...

I0408 23:23:25.731975 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618604935-6159e179538c4-18274eba-17f55b38
I0408 23:23:38.847539 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.04_create_forwarding_rule ---
I0408 23:23:38.847747 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.05_start_test_servers ---
I0408 23:23:38.847903 125727502180352 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240408-2322-8ocqv: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0408 23:23:38.848145 125727502180352 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2322-8ocqv%22;timeRange=2024-04-08T23:23:38.848015Z%2F2024-04-08T23:53:38.848015Z?project=grpc-testing
I0408 23:23:38.852051 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240408-2322-8ocqv
labels:
name: psm-interop-server-20240408-2322-8ocqv
owner: xds-k8s-interop-test
...

I0408 23:23:38.879044 125727502180352 k8s_base_runner.py:367] Namespace psm-interop-server-20240408-2322-8ocqv created
I0408 23:23:38.881489 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2322-8ocqv
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2322-8ocqv-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:23:38.912341 125727502180352 k8s_base_runner.py:367] Service psm-grpc-server created
I0408 23:23:38.912563 125727502180352 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240408-2322-8ocqv
I0408 23:23:38.942267 125727502180352 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240408-2322-8ocqv: neg_name=psm-interop-server-20240408-2322-8ocqv-psm-grpc-server, port=8080, zones=['us-central1-a']
I0408 23:23:38.942460 125727502180352 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2322-8ocqv/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:23:40.717417 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2322-8ocqv
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:23:40.737052 125727502180352 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0408 23:23:40.747652 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2322-8ocqv
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0wkgk7no
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0wkgk7no
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0wkgk7no
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2322-8ocqv-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:23:40.784031 125727502180352 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0408 23:23:40.784192 125727502180352 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0408 23:23:41.856064 125727502180352 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-7d5c6bfb7f-fjdmd', 'psm-grpc-server-7d5c6bfb7f-ljm2w', 'psm-grpc-server-7d5c6bfb7f-md5wp']
I0408 23:23:41.856397 125727502180352 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-7d5c6bfb7f-fjdmd to start
I0408 23:23:44.924587 125727502180352 k8s_base_runner.py:985] Pod psm-grpc-server-7d5c6bfb7f-fjdmd ready, IP: 10.120.11.157
I0408 23:23:44.924826 125727502180352 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-7d5c6bfb7f-fjdmd to affinity_test/test_app_logs/psm-interop-server-20240408-2322-8ocqv_psm-grpc-server-7d5c6bfb7f-fjdmd.log
I0408 23:23:44.925341 125727406282304 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2322-8ocqv] Starting log collection thread 125727406282304 for psm-grpc-server-7d5c6bfb7f-fjdmd
I0408 23:23:44.925559 125727502180352 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-7d5c6bfb7f-ljm2w to start
I0408 23:23:44.968137 125727502180352 k8s_base_runner.py:985] Pod psm-grpc-server-7d5c6bfb7f-ljm2w ready, IP: 10.120.10.209
I0408 23:23:44.968379 125727502180352 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-7d5c6bfb7f-ljm2w to affinity_test/test_app_logs/psm-interop-server-20240408-2322-8ocqv_psm-grpc-server-7d5c6bfb7f-ljm2w.log
I0408 23:23:44.968859 125727395849792 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2322-8ocqv] Starting log collection thread 125727395849792 for psm-grpc-server-7d5c6bfb7f-ljm2w
I0408 23:23:44.969068 125727502180352 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-7d5c6bfb7f-md5wp to start
I0408 23:23:45.002659 125727502180352 k8s_base_runner.py:985] Pod psm-grpc-server-7d5c6bfb7f-md5wp ready, IP: 10.120.17.84
I0408 23:23:45.002870 125727502180352 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-7d5c6bfb7f-md5wp to affinity_test/test_app_logs/psm-interop-server-20240408-2322-8ocqv_psm-grpc-server-7d5c6bfb7f-md5wp.log
I0408 23:23:45.003225 125727315850816 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2322-8ocqv] Starting log collection thread 125727315850816 for psm-grpc-server-7d5c6bfb7f-md5wp
I0408 23:23:45.003307 125727502180352 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0408 23:23:49.097680 125727502180352 k8s_base_runner.py:953] Deployment psm-grpc-server has 3 replicas available
I0408 23:23:49.097928 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.05_start_test_servers ---
I0408 23:23:49.097997 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0408 23:23:49.105932 125727502180352 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:23:49.173487 125727502180352 traffic_director.py:259] Loaded NEG "psm-interop-server-20240408-2322-8ocqv-psm-grpc-server" in zone us-central1-a
I0408 23:23:49.173676 125727502180352 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240408-2322-8ocqv: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2322-8ocqv-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2322-8ocqv-psm-grpc-server', zone='us-central1-a')}
I0408 23:23:49.176891 125727502180352 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2322-8ocqv-psm-grpc-server
maxRatePerEndpoint: 5
...

I0408 23:23:50.348270 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618629183-6159e19073694-1ca4a4ec-a28a4fbd
I0408 23:24:46.336769 125727502180352 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240408-2322-8ocqv to report all backends healthy: ['psm-interop-server-20240408-2322-8ocqv-psm-grpc-server']
I0408 23:24:59.582370 125727502180352 compute.py:494] Backend psm-interop-server-20240408-2322-8ocqv-psm-grpc-server in zone us-central1-a reported healthy
I0408 23:25:03.587021 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0408 23:25:03.587260 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.07_start_test_client ---
I0408 23:25:03.587365 125727502180352 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240408-2322-8ocqv: server_target=xds:///psm-grpc-server:54704 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0408 23:25:03.587620 125727502180352 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2322-8ocqv%22;timeRange=2024-04-08T23:25:03.587504Z%2F2024-04-08T23:55:03.587504Z?project=grpc-testing
I0408 23:25:03.589200 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240408-2322-8ocqv
labels:
name: psm-interop-client-20240408-2322-8ocqv
owner: xds-k8s-interop-test
...

I0408 23:25:03.635185 125727502180352 k8s_base_runner.py:367] Namespace psm-interop-client-20240408-2322-8ocqv created
I0408 23:25:03.635528 125727502180352 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2322-8ocqv/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:25:04.096015 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2322-8ocqv
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:25:04.131111 125727502180352 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0408 23:25:04.138274 125727502180352 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2322-8ocqv
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qnbp84u4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qnbp84u4
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qnbp84u4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:54704"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0408 23:25:04.173623 125727502180352 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0408 23:25:04.173927 125727502180352 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0408 23:25:04.210216 125727502180352 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-78964664c-69wdx']
I0408 23:25:04.210441 125727502180352 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-78964664c-69wdx to start
I0408 23:25:24.474309 125727502180352 k8s_base_runner.py:985] Pod psm-grpc-client-78964664c-69wdx ready, IP: 10.120.9.35
I0408 23:25:24.474588 125727502180352 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-78964664c-69wdx to affinity_test/test_app_logs/psm-interop-client-20240408-2322-8ocqv_psm-grpc-client-78964664c-69wdx.log
I0408 23:25:24.475137 125727307458112 k8s_log_collector.py:71] [ns/psm-interop-client-20240408-2322-8ocqv] Starting log collection thread 125727307458112 for psm-grpc-client-78964664c-69wdx
I0408 23:25:24.475324 125727502180352 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0408 23:25:26.537351 125727502180352 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0408 23:25:26.537765 125727502180352 client_app.py:292] [psm-grpc-client-78964664c-69wdx] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0408 23:25:26.580229 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:25:36.602657 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:25:46.623051 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:25:56.643655 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:26:06.666253 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:26:22.693336 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:26:47.730547 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:27:12.766039 125727502180352 client_app.py:320] [psm-grpc-client-78964664c-69wdx] xDS control plane channel:
I0408 23:27:14.775923 125727502180352 client_app.py:331] [psm-grpc-client-78964664c-69wdx] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0408 23:27:14.776122 125727502180352 client_app.py:302] [psm-grpc-client-78964664c-69wdx] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0408 23:27:14.776318 125727502180352 client_app.py:252] [psm-grpc-client-78964664c-69wdx] Waiting to report a READY channel to xds:///psm-grpc-server:54704
I0408 23:27:14.780377 125727502180352 client_app.py:379] [psm-grpc-client-78964664c-69wdx] Server channel:
I0408 23:27:14.787969 125727502180352 client_app.py:392] [psm-grpc-client-78964664c-69wdx] Found subchannel in state READY:
I0408 23:27:14.788337 125727502180352 client_app.py:263] [psm-grpc-client-78964664c-69wdx] Channel to xds:///psm-grpc-server:54704 transitioned to state READY:
I0408 23:27:14.788912 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:27:14.804695 125727502180352 affinity_test.py:105] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/719df673-5666-44d5-8d76-b042a0968bd8",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.9.35",
"TRAFFICDIRECTOR_DIRECTPATH_C2P_IPV6_CAPABLE": true,
"TRAFFICDIRECTOR_GRPC_BOOTSTRAP_GENERATOR_SHA": "2bf1b5ed00f852ffea8d24759c6fa673acc9ef10"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.9.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:54704",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:54704",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-04-08T23:26:48Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704",
"virtualHosts": [
{
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704-route-0"
}
]
}
]
},
"lastUpdated": "2024-04-08T23:26:48.039Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240408-2322-8ocqv"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240408-2322-8ocqv",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-04-08T23:26:48.074Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"versionInfo": "1",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3625774689591870100_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.84",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.10.209",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.11.157",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-04-08T23:26:49.659Z",
"clientStatus": "ACKED"
}
]
}
I0408 23:27:14.805490 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.07_start_test_client ---
I0408 23:27:14.805662 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0408 23:27:14.805792 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:27:14.810609 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0408 23:27:14.810873 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0408 23:27:14.811172 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:27:18.859219 125727502180352 xds_k8s_testcase.py:669] [psm-grpc-client-78964664c-69wdx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d5c6bfb7f-fjdmd: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-7d5c6bfb7f-fjdmd: 100

I0408 23:27:18.859603 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0408 23:27:18.859788 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0408 23:27:18.859943 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:27:22.914433 125727502180352 xds_k8s_testcase.py:669] [psm-grpc-client-78964664c-69wdx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d5c6bfb7f-fjdmd: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-7d5c6bfb7f-fjdmd: 100

I0408 23:27:22.924509 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0408 23:27:22.924791 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0408 23:27:22.924931 125727502180352 server_app.py:108] [psm-grpc-server-7d5c6bfb7f-fjdmd] >> Setting health status to NOT_SERVING
I0408 23:27:22.925247 125727502180352 grpc.py:75] [psm-grpc-server-7d5c6bfb7f-fjdmd:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), wait_for_ready=True, timeout=90)
I0408 23:27:22.933584 125727502180352 grpc.py:75] [psm-grpc-server-7d5c6bfb7f-fjdmd:8080] >> RPC Health.Check(request=HealthCheckRequest({}), wait_for_ready=True, timeout=90)
I0408 23:27:22.935635 125727502180352 server_app.py:112] [psm-grpc-server-7d5c6bfb7f-fjdmd] << Health status status: NOT_SERVING

I0408 23:27:22.935917 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0408 23:27:22.936043 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0408 23:27:22.936219 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:27:22.941148 125727502180352 affinity_test.py:166] CSDS got unexpected endpoints, will retry after 10 seconds
I0408 23:27:32.951572 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:27:32.958323 125727502180352 affinity_test.py:179] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/719df673-5666-44d5-8d76-b042a0968bd8",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.9.35",
"TRAFFICDIRECTOR_DIRECTPATH_C2P_IPV6_CAPABLE": true,
"TRAFFICDIRECTOR_GRPC_BOOTSTRAP_GENERATOR_SHA": "2bf1b5ed00f852ffea8d24759c6fa673acc9ef10"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.9.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:54704",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:54704",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-04-08T23:26:48Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704",
"virtualHosts": [
{
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2322-8ocqv_0_psm-grpc-server:54704-route-0"
}
]
}
]
},
"lastUpdated": "2024-04-08T23:26:48.039Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"versionInfo": "1712618797299837082",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240408-2322-8ocqv"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240408-2322-8ocqv",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-04-08T23:26:48.074Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"versionInfo": "5",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_4767512301223396005",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3625774689591870100_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.84",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.10.209",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.11.157",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-04-08T23:27:29.659Z",
"clientStatus": "ACKED"
}
]
}
I0408 23:27:32.958992 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0408 23:27:32.959088 125727502180352 xds_k8s_testcase.py:299] --- Starting subTest AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0408 23:27:32.959208 125727502180352 grpc.py:75] [psm-grpc-client-78964664c-69wdx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:27:36.985011 125727502180352 xds_k8s_testcase.py:669] [psm-grpc-client-78964664c-69wdx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7d5c6bfb7f-ljm2w: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-7d5c6bfb7f-ljm2w: 100

I0408 23:27:36.985417 125727502180352 xds_k8s_testcase.py:304] --- Finished subTest AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0408 23:27:36.985680 125727502180352 xds_k8s_testcase.py:755] ----- TestMethod AffinityTest.test_affinity teardown -----
I0408 23:27:37.078392 125727502180352 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2322-8ocqv"
I0408 23:27:37.802992 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618857082-6159e269cacfa-bb777737-670d8828
I0408 23:28:01.842991 125727502180352 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240408-2322-8ocqv"
I0408 23:28:01.929627 125727502180352 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240408-2322-8ocqv"
I0408 23:28:02.359946 125727502180352 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2322-8ocqv"
I0408 23:28:02.795490 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618882362-6159e281e6de2-19a92f10-378514f7
I0408 23:28:05.082421 125727502180352 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240408-2322-8ocqv"
I0408 23:28:05.429839 125727502180352 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2322-8ocqv"
I0408 23:28:05.992823 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618885438-6159e284d5a61-0b364f11-cf4d747e
I0408 23:28:08.291095 125727502180352 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240408-2322-8ocqv"
I0408 23:28:08.668940 125727502180352 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240408-2322-8ocqv"
I0408 23:28:09.301053 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618888677-6159e287ec895-c63a61fc-56634438
I0408 23:28:37.428123 125727502180352 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240408-2322-8ocqv"
I0408 23:28:37.761738 125727502180352 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240408-2322-8ocqv"
I0408 23:28:38.180777 125727502180352 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240408-2322-8ocqv"
I0408 23:28:38.718374 125727502180352 compute.py:614] Waiting 600 sec for compute operation id: operation-1712618918263-6159e2a423c44-8aa56f5c-c82a7dd7
I0408 23:28:43.172164 125727502180352 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0408 23:29:23.307233 125727502180352 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0408 23:29:23.307553 125727502180352 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2322-8ocqv/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:29:23.785823 125727502180352 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0408 23:29:23.829598 125727502180352 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0408 23:29:23.829827 125727502180352 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240408-2322-8ocqv
I0408 23:29:53.952919 125727502180352 k8s_base_runner.py:941] Namespace psm-interop-client-20240408-2322-8ocqv deleted
I0408 23:29:53.953093 125727502180352 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0408 23:30:04.036028 125727502180352 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0408 23:30:04.036226 125727502180352 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0408 23:30:05.116360 125727502180352 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0408 23:30:05.116588 125727502180352 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2322-8ocqv/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:30:05.640075 125727502180352 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0408 23:30:05.691056 125727502180352 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0408 23:30:05.691244 125727502180352 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2322-8ocqv
I0408 23:32:05.935920 125727502180352 k8s_base_runner.py:941] Namespace psm-interop-server-20240408-2322-8ocqv deleted
I0408 23:32:05.936212 125727502180352 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0408 23:32:05.936406 125727502180352 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-qnbp84u4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2322-8ocqv%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-qnbp84u4%22;timeRange=2024-04-08T23:25:03.587497Z%2F2024-04-08T23:29:53.953066Z;cursorTimestamp=2024-04-08T23:25:26.537578Z?project=grpc-testing
I0408 23:32:05.936513 125727502180352 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-0wkgk7no:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2322-8ocqv%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-0wkgk7no%22;timeRange=2024-04-08T23:23:38.848008Z%2F2024-04-08T23:32:05.936074Z;cursorTimestamp=2024-04-08T23:23:49.097875Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
I0408 23:32:05.936685 125727502180352 base_testcase.py:67] ----- PSM Test Case PASSED: AffinityTest.test_affinity -----

----------------------------------------------------------------------
Ran 1 test in 559.906s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/api_listener_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0408 23:32:06.961091 139924700291072 xds_k8s_testcase.py:162] ----- Testing ApiListenerTest -----
I0408 23:32:06.961198 139924700291072 xds_k8s_testcase.py:163] Logs timezone: UTC
I0408 23:32:06.961375 139924700291072 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0408 23:32:06.973535 139924700291072 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0408 23:32:06.985710 139924700291072 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0408 23:32:06.989299 139924700291072 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240408-2332-fv885
I0408 23:32:07.226691 139924700291072 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/api_listener_test
I0408 23:32:07.527873 139924700291072 xds_k8s_testcase.py:731] Found unused xds port: 12765
I0408 23:32:07.528189 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.00_create_health_check ---
I0408 23:32:07.528283 139924700291072 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240408-2332-fv885"
I0408 23:32:07.530618 139924700291072 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240408-2332-fv885
type: GRPC
...

I0408 23:32:08.510375 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619127595-6159e36bc6054-ffebd618-35ddffbc
I0408 23:32:10.816594 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.00_create_health_check ---
I0408 23:32:10.816791 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.01_create_backend_services ---
I0408 23:32:10.816901 139924700291072 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240408-2332-fv885"
I0408 23:32:10.822338 139924700291072 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2332-fv885
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240408-2332-fv885
protocol: GRPC
...

I0408 23:32:11.452143 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619130824-6159e36eda678-1f586b72-7b2634a2
I0408 23:32:50.409644 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.01_create_backend_services ---
I0408 23:32:50.409871 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0408 23:32:50.409988 139924700291072 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240408-2332-fv885": psm-grpc-server:12765 -> psm-interop-backend-service-20240408-2332-fv885
I0408 23:32:50.417124 139924700291072 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2332-fv885
hostRules:
- hosts:
- psm-grpc-server:12765
pathMatcher: psm-interop-path-matcher-20240408-2332-fv885
name: psm-interop-url-map-20240408-2332-fv885
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2332-fv885
name: psm-interop-path-matcher-20240408-2332-fv885
...

I0408 23:32:51.158432 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619170419-6159e3949d459-a2fd93f5-4282b742
I0408 23:32:53.500233 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0408 23:32:53.500436 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0408 23:32:53.500539 139924700291072 traffic_director.py:578] Creating target psm-interop-target-proxy-20240408-2332-fv885 proxy "GRPC" to URL map psm-interop-url-map-20240408-2332-fv885
I0408 23:32:53.502019 139924700291072 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240408-2332-fv885
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240408-2332-fv885
validate_for_proxyless: true
...

I0408 23:32:54.062019 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619173508-6159e3978f42d-dd0ef77b-ca833983
I0408 23:32:56.403116 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0408 23:32:56.403300 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0408 23:32:56.403408 139924700291072 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240408-2332-fv885" in network "default-vpc": 0.0.0.0:12765 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2332-fv885
I0408 23:32:56.405243 139924700291072 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240408-2332-fv885
network: global/networks/default-vpc
portRange: 12765
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2332-fv885
...

I0408 23:32:57.303690 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619176411-6159e39a542a8-f50c02fe-3150b41d
I0408 23:33:10.420064 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0408 23:33:10.420265 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.05_start_test_server ---
I0408 23:33:10.420366 139924700291072 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240408-2332-fv885: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0408 23:33:10.420596 139924700291072 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2332-fv885%22;timeRange=2024-04-08T23:33:10.420472Z%2F2024-04-09T00:03:10.420472Z?project=grpc-testing
I0408 23:33:10.424353 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240408-2332-fv885
labels:
name: psm-interop-server-20240408-2332-fv885
owner: xds-k8s-interop-test
...

I0408 23:33:10.456415 139924700291072 k8s_base_runner.py:367] Namespace psm-interop-server-20240408-2332-fv885 created
I0408 23:33:10.458823 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2332-fv885
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2332-fv885-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:33:10.496643 139924700291072 k8s_base_runner.py:367] Service psm-grpc-server created
I0408 23:33:10.496824 139924700291072 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240408-2332-fv885
I0408 23:33:10.525182 139924700291072 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240408-2332-fv885: neg_name=psm-interop-server-20240408-2332-fv885-psm-grpc-server, port=8080, zones=['us-central1-a']
I0408 23:33:10.525314 139924700291072 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2332-fv885/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:33:11.061057 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2332-fv885
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:33:11.082729 139924700291072 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0408 23:33:11.094156 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2332-fv885
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8v6dim2l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8v6dim2l
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8v6dim2l
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2332-fv885-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:33:11.134993 139924700291072 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0408 23:33:11.135188 139924700291072 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0408 23:33:12.176069 139924700291072 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-84556dcc88-r5nnq']
I0408 23:33:12.176264 139924700291072 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-84556dcc88-r5nnq to start
I0408 23:33:14.237040 139924700291072 k8s_base_runner.py:985] Pod psm-grpc-server-84556dcc88-r5nnq ready, IP: 10.120.9.37
I0408 23:33:14.237267 139924700291072 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-84556dcc88-r5nnq to api_listener_test/test_app_logs/psm-interop-server-20240408-2332-fv885_psm-grpc-server-84556dcc88-r5nnq.log
I0408 23:33:14.237788 139924603131456 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2332-fv885] Starting log collection thread 139924603131456 for psm-grpc-server-84556dcc88-r5nnq
I0408 23:33:14.237961 139924700291072 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0408 23:33:19.323543 139924700291072 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0408 23:33:19.323821 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.05_start_test_server ---
I0408 23:33:19.323920 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0408 23:33:19.328875 139924700291072 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:33:19.416099 139924700291072 traffic_director.py:259] Loaded NEG "psm-interop-server-20240408-2332-fv885-psm-grpc-server" in zone us-central1-a
I0408 23:33:19.416322 139924700291072 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240408-2332-fv885: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2332-fv885-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2332-fv885-psm-grpc-server', zone='us-central1-a')}
I0408 23:33:19.419842 139924700291072 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2332-fv885-psm-grpc-server
maxRatePerEndpoint: 5
...

I0408 23:33:20.464190 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619199422-6159e3b045ec1-0bd295f7-ebfc5a68
I0408 23:34:16.689317 139924700291072 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240408-2332-fv885 to report all backends healthy: ['psm-interop-server-20240408-2332-fv885-psm-grpc-server']
I0408 23:34:25.203263 139924700291072 compute.py:494] Backend psm-interop-server-20240408-2332-fv885-psm-grpc-server in zone us-central1-a reported healthy
I0408 23:34:29.207877 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0408 23:34:29.208087 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.07_start_test_client ---
I0408 23:34:29.208200 139924700291072 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240408-2332-fv885: server_target=xds:///psm-grpc-server:12765 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0408 23:34:29.208436 139924700291072 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2332-fv885%22;timeRange=2024-04-08T23:34:29.208328Z%2F2024-04-09T00:04:29.208328Z?project=grpc-testing
I0408 23:34:29.210017 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240408-2332-fv885
labels:
name: psm-interop-client-20240408-2332-fv885
owner: xds-k8s-interop-test
...

I0408 23:34:29.256157 139924700291072 k8s_base_runner.py:367] Namespace psm-interop-client-20240408-2332-fv885 created
I0408 23:34:29.256365 139924700291072 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2332-fv885/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:34:29.720261 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2332-fv885
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:34:29.747797 139924700291072 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0408 23:34:29.754420 139924700291072 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2332-fv885
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zmtkwyli
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zmtkwyli
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zmtkwyli
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:12765"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0408 23:34:29.792459 139924700291072 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0408 23:34:29.792670 139924700291072 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0408 23:34:29.825291 139924700291072 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5d94fcd9cc-2v8md']
I0408 23:34:29.825450 139924700291072 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-5d94fcd9cc-2v8md to start
I0408 23:34:51.078815 139924700291072 k8s_base_runner.py:985] Pod psm-grpc-client-5d94fcd9cc-2v8md ready, IP: 10.120.15.56
I0408 23:34:51.079030 139924700291072 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-5d94fcd9cc-2v8md to api_listener_test/test_app_logs/psm-interop-client-20240408-2332-fv885_psm-grpc-client-5d94fcd9cc-2v8md.log
I0408 23:34:51.079521 139924594738752 k8s_log_collector.py:71] [ns/psm-interop-client-20240408-2332-fv885] Starting log collection thread 139924594738752 for psm-grpc-client-5d94fcd9cc-2v8md
I0408 23:34:51.079694 139924700291072 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0408 23:34:55.157674 139924700291072 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0408 23:34:55.158046 139924700291072 client_app.py:292] [psm-grpc-client-5d94fcd9cc-2v8md] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0408 23:34:55.197788 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:05.221200 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:15.242450 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:25.264065 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:35.287694 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:51.313736 139924700291072 client_app.py:320] [psm-grpc-client-5d94fcd9cc-2v8md] xDS control plane channel:
I0408 23:35:53.324700 139924700291072 client_app.py:331] [psm-grpc-client-5d94fcd9cc-2v8md] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0408 23:35:53.324954 139924700291072 client_app.py:302] [psm-grpc-client-5d94fcd9cc-2v8md] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0408 23:35:53.325211 139924700291072 client_app.py:252] [psm-grpc-client-5d94fcd9cc-2v8md] Waiting to report a READY channel to xds:///psm-grpc-server:12765
I0408 23:35:53.328994 139924700291072 client_app.py:379] [psm-grpc-client-5d94fcd9cc-2v8md] Server channel:
I0408 23:35:53.337075 139924700291072 client_app.py:392] [psm-grpc-client-5d94fcd9cc-2v8md] Found subchannel in state READY:
I0408 23:35:53.337385 139924700291072 client_app.py:263] [psm-grpc-client-5d94fcd9cc-2v8md] Channel to xds:///psm-grpc-server:12765 transitioned to state READY:
I0408 23:35:53.337554 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.07_start_test_client ---
I0408 23:35:53.337636 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0408 23:35:53.337841 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:35:53.352618 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0408 23:35:53.352874 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0408 23:35:53.353045 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:35:57.405490 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:35:57.405815 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0408 23:35:57.405958 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0408 23:35:57.406052 139924700291072 traffic_director.py:540] Creating alternative URL map "psm-interop-url-map-alt-20240408-2332-fv885": psm-grpc-server:12765 -> psm-interop-backend-service-20240408-2332-fv885
I0408 23:35:57.409482 139924700291072 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2332-fv885
hostRules:
- hosts:
- psm-grpc-server:12765
pathMatcher: psm-interop-path-matcher-20240408-2332-fv885
name: psm-interop-url-map-alt-20240408-2332-fv885
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2332-fv885
name: psm-interop-path-matcher-20240408-2332-fv885
...

I0408 23:35:58.010047 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619357411-6159e446f1a57-007c80a7-6479e371
I0408 23:36:00.349357 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0408 23:36:00.349544 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0408 23:36:00.349631 139924700291072 traffic_director.py:613] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20240408-2332-fv885" to URL map psm-interop-url-map-alt-20240408-2332-fv885
I0408 23:36:00.350952 139924700291072 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20240408-2332-fv885
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20240408-2332-fv885
validate_for_proxyless: false
...

I0408 23:36:01.002065 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619360352-6159e449bfa3c-fb821a03-3b0a942e
I0408 23:36:03.361361 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0408 23:36:03.361562 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0408 23:36:03.361661 139924700291072 traffic_director.py:683] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20240408-2332-fv885" in network "default-vpc": 10.10.10.10:12765 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240408-2332-fv885
I0408 23:36:03.363599 139924700291072 compute.py:537] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20240408-2332-fv885
network: global/networks/default-vpc
portRange: 12765
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240408-2332-fv885
...

I0408 23:36:04.241072 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619363365-6159e44c9f428-3df9d63c-1c00d25c
I0408 23:36:17.421591 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0408 23:36:17.421848 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0408 23:36:17.422022 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:36:21.487627 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:36:21.487965 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:36:21.492548 139924700291072 api_listener_test.py:105] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/fa0448d6-c1ed-4007-a047-8a5da70a9a82",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.15.56",
"TRAFFICDIRECTOR_DIRECTPATH_C2P_IPV6_CAPABLE": true,
"TRAFFICDIRECTOR_GRPC_BOOTSTRAP_GENERATOR_SHA": "2bf1b5ed00f852ffea8d24759c6fa673acc9ef10"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Node Pure JS",
"userAgentVersion": "1.9.2",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning"
]
},
"genericXdsConfigs": [
{
"typeUrl": "envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:12765",
"versionInfo": "1712619309573941230",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:12765",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240408-2332-fv885_0_psm-grpc-server:12765"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-04-08T23:35:42.353Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2332-fv885_0_psm-grpc-server:12765",
"versionInfo": "1712619309573941230",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2332-fv885_0_psm-grpc-server:12765",
"virtualHosts": [
{
"domains": [
"*"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_8931035614937727124",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240408-2332-fv885_0_psm-grpc-server:12765-route-0"
}
]
}
]
},
"lastUpdated": "2024-04-08T23:35:42.381Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8931035614937727124",
"versionInfo": "1712619309573941230",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8931035614937727124",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240408-2332-fv885"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240408-2332-fv885",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-04-08T23:35:42.411Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_8931035614937727124",
"versionInfo": "1",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_8931035614937727124",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_6112025261748371545_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.9.37",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-04-08T23:35:44.560Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1712619309573941230
I0408 23:36:21.493090 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0408 23:36:21.493198 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0408 23:36:21.493463 139924700291072 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2332-fv885"
I0408 23:36:22.321007 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619381496-6159e45de9acb-27d5359d-920ae893
I0408 23:36:31.112486 139924700291072 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2332-fv885"
I0408 23:36:31.603196 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619391119-6159e46717088-8453733b-f17a76a1
I0408 23:36:34.026411 139924700291072 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2332-fv885"
I0408 23:36:34.671252 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619394035-6159e469deef5-aae6de1c-6ea2d415
I0408 23:36:36.980165 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0408 23:36:36.980357 139924700291072 xds_k8s_testcase.py:299] --- Starting subTest ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0408 23:36:36.980700 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:36:41.000428 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:36:41.000815 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:36:41.004542 139924700291072 xds_k8s_testcase.py:584] Routing config not propagated yet. Retrying.
I0408 23:36:41.004950 139924700291072 retryers.py:213] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps
I0408 23:36:43.007303 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:36:47.049960 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:36:47.050315 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:36:47.054461 139924700291072 xds_k8s_testcase.py:584] Routing config not propagated yet. Retrying.
I0408 23:36:47.054646 139924700291072 retryers.py:213] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps
I0408 23:36:49.056930 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:36:53.094702 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:36:53.095176 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:36:53.100279 139924700291072 xds_k8s_testcase.py:584] Routing config not propagated yet. Retrying.
I0408 23:36:53.100608 139924700291072 retryers.py:213] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps
I0408 23:36:55.102978 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:36:59.097707 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:36:59.097992 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:36:59.102533 139924700291072 xds_k8s_testcase.py:584] Routing config not propagated yet. Retrying.
I0408 23:36:59.102746 139924700291072 retryers.py:213] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps
I0408 23:37:01.105049 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:37:05.160686 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:37:05.161063 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0408 23:37:05.165758 139924700291072 grpc.py:75] [psm-grpc-client-5d94fcd9cc-2v8md:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0408 23:37:09.204044 139924700291072 xds_k8s_testcase.py:669] [psm-grpc-client-5d94fcd9cc-2v8md] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-84556dcc88-r5nnq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-84556dcc88-r5nnq: 100

I0408 23:37:09.204339 139924700291072 xds_k8s_testcase.py:593] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1712619410527515200
I0408 23:37:09.204565 139924700291072 xds_k8s_testcase.py:304] --- Finished subTest ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0408 23:37:09.204688 139924700291072 xds_k8s_testcase.py:755] ----- TestMethod ApiListenerTest.test_api_listener teardown -----
I0408 23:37:09.286122 139924700291072 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2332-fv885"
I0408 23:37:09.441784 139924700291072 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240408-2332-fv885"
I0408 23:37:10.185304 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619429445-6159e48ba3e39-d1c7c426-983993d8
I0408 23:37:23.252361 139924700291072 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240408-2332-fv885"
I0408 23:37:23.674025 139924700291072 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2332-fv885"
I0408 23:37:24.049903 139924700291072 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240408-2332-fv885"
I0408 23:37:24.672287 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619444057-6159e49993415-a59550cf-d6df77de
I0408 23:37:26.980952 139924700291072 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2332-fv885"
I0408 23:37:27.411903 139924700291072 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240408-2332-fv885"
I0408 23:37:28.173646 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619447420-6159e49cc8691-5af4987c-f980392f
I0408 23:37:30.490735 139924700291072 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240408-2332-fv885"
I0408 23:37:31.137508 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619450499-6159e49fb8062-aaeea8f7-1edd92e8
I0408 23:37:59.305696 139924700291072 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240408-2332-fv885"
I0408 23:37:59.678072 139924700291072 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240408-2332-fv885"
I0408 23:38:00.110297 139924700291072 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240408-2332-fv885"
I0408 23:38:00.706920 139924700291072 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619480187-6159e4bc08174-01908cdb-148d2129
I0408 23:38:05.175653 139924700291072 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0408 23:38:45.302064 139924700291072 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0408 23:38:45.302306 139924700291072 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2332-fv885/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:38:45.780287 139924700291072 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0408 23:38:45.833538 139924700291072 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0408 23:38:45.833708 139924700291072 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240408-2332-fv885
I0408 23:39:15.949809 139924700291072 k8s_base_runner.py:941] Namespace psm-interop-client-20240408-2332-fv885 deleted
I0408 23:39:15.949981 139924700291072 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0408 23:39:26.032570 139924700291072 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0408 23:39:26.032801 139924700291072 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0408 23:39:27.085709 139924700291072 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0408 23:39:27.085924 139924700291072 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2332-fv885/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:39:27.571784 139924700291072 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0408 23:39:27.621969 139924700291072 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0408 23:39:27.622121 139924700291072 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2332-fv885
I0408 23:40:27.749313 139924700291072 k8s_base_runner.py:941] Namespace psm-interop-server-20240408-2332-fv885 deleted
I0408 23:40:27.749643 139924700291072 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0408 23:40:27.749846 139924700291072 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-zmtkwyli:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2332-fv885%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-zmtkwyli%22;timeRange=2024-04-08T23:34:29.208320Z%2F2024-04-08T23:39:15.949957Z;cursorTimestamp=2024-04-08T23:34:55.157879Z?project=grpc-testing
I0408 23:40:27.749994 139924700291072 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-8v6dim2l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2332-fv885%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-8v6dim2l%22;timeRange=2024-04-08T23:33:10.420466Z%2F2024-04-08T23:40:27.749510Z;cursorTimestamp=2024-04-08T23:33:19.323754Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
I0408 23:40:27.750227 139924700291072 base_testcase.py:67] ----- PSM Test Case PASSED: ApiListenerTest.test_api_listener -----

----------------------------------------------------------------------
Ran 1 test in 500.791s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/baseline_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0408 23:40:28.744717 131612553863168 xds_k8s_testcase.py:162] ----- Testing BaselineTest -----
I0408 23:40:28.744829 131612553863168 xds_k8s_testcase.py:163] Logs timezone: UTC
I0408 23:40:28.744995 131612553863168 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0408 23:40:28.757537 131612553863168 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0408 23:40:28.770583 131612553863168 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] BaselineTest.test_traffic_director_grpc_setup
I0408 23:40:28.774015 131612553863168 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240408-2340-vugso
I0408 23:40:29.134621 131612553863168 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/baseline_test
I0408 23:40:29.412254 131612553863168 xds_k8s_testcase.py:731] Found unused xds port: 19210
I0408 23:40:29.412516 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0408 23:40:29.412578 131612553863168 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240408-2340-vugso"
I0408 23:40:29.414676 131612553863168 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240408-2340-vugso
type: GRPC
...

I0408 23:40:30.168023 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619629489-6159e54a6acf1-b0cb6f56-da9ffe7d
I0408 23:40:34.655668 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0408 23:40:34.655850 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0408 23:40:34.655958 131612553863168 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240408-2340-vugso"
I0408 23:40:34.661333 131612553863168 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2340-vugso
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240408-2340-vugso
protocol: GRPC
...

I0408 23:40:35.384547 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619634667-6159e54f5b015-8d0a0b71-c91132dd
I0408 23:41:03.610094 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0408 23:41:03.610307 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0408 23:41:03.610402 131612553863168 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240408-2340-vugso": psm-grpc-server:19210 -> psm-interop-backend-service-20240408-2340-vugso
I0408 23:41:03.617947 131612553863168 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2340-vugso
hostRules:
- hosts:
- psm-grpc-server:19210
pathMatcher: psm-interop-path-matcher-20240408-2340-vugso
name: psm-interop-url-map-20240408-2340-vugso
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2340-vugso
name: psm-interop-path-matcher-20240408-2340-vugso
...

I0408 23:41:04.151370 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619663619-6159e56af775e-0d2a468c-99db9129
I0408 23:41:06.510895 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0408 23:41:06.511090 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0408 23:41:06.511188 131612553863168 traffic_director.py:578] Creating target psm-interop-target-proxy-20240408-2340-vugso proxy "GRPC" to URL map psm-interop-url-map-20240408-2340-vugso
I0408 23:41:06.512633 131612553863168 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240408-2340-vugso
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240408-2340-vugso
validate_for_proxyless: true
...

I0408 23:41:07.035863 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619666514-6159e56dba29b-7daae8c7-6889a219
I0408 23:41:09.333907 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0408 23:41:09.334104 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0408 23:41:09.334212 131612553863168 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240408-2340-vugso" in network "default-vpc": 0.0.0.0:19210 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2340-vugso
I0408 23:41:09.336080 131612553863168 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240408-2340-vugso
network: global/networks/default-vpc
portRange: 19210
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2340-vugso
...

I0408 23:41:10.278103 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619669338-6159e5706b7f9-10cb1ce4-bc039f01
I0408 23:41:23.280447 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0408 23:41:23.280665 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0408 23:41:23.280772 131612553863168 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240408-2340-vugso: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0408 23:41:23.280995 131612553863168 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2340-vugso%22;timeRange=2024-04-08T23:41:23.280877Z%2F2024-04-09T00:11:23.280877Z?project=grpc-testing
I0408 23:41:23.284749 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240408-2340-vugso
labels:
name: psm-interop-server-20240408-2340-vugso
owner: xds-k8s-interop-test
...

I0408 23:41:23.338352 131612553863168 k8s_base_runner.py:367] Namespace psm-interop-server-20240408-2340-vugso created
I0408 23:41:23.340617 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2340-vugso
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2340-vugso-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:41:23.381662 131612553863168 k8s_base_runner.py:367] Service psm-grpc-server created
I0408 23:41:23.381884 131612553863168 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240408-2340-vugso
I0408 23:41:23.411201 131612553863168 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240408-2340-vugso: neg_name=psm-interop-server-20240408-2340-vugso-psm-grpc-server, port=8080, zones=['us-central1-a']
I0408 23:41:23.411399 131612553863168 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2340-vugso/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:41:23.830195 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2340-vugso
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:41:23.852979 131612553863168 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0408 23:41:23.864763 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2340-vugso
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-v849g28l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-v849g28l
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-v849g28l
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2340-vugso-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:41:23.904211 131612553863168 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0408 23:41:23.904410 131612553863168 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0408 23:41:24.944949 131612553863168 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5f8456c994-sz8nq']
I0408 23:41:24.945201 131612553863168 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-5f8456c994-sz8nq to start
I0408 23:41:28.016146 131612553863168 k8s_base_runner.py:985] Pod psm-grpc-server-5f8456c994-sz8nq ready, IP: 10.120.10.211
I0408 23:41:28.016357 131612553863168 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-5f8456c994-sz8nq to baseline_test/test_app_logs/psm-interop-server-20240408-2340-vugso_psm-grpc-server-5f8456c994-sz8nq.log
I0408 23:41:28.016857 131612457293376 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2340-vugso] Starting log collection thread 131612457293376 for psm-grpc-server-5f8456c994-sz8nq
I0408 23:41:28.017173 131612553863168 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0408 23:41:33.092872 131612553863168 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0408 23:41:33.093099 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0408 23:41:33.093187 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0408 23:41:33.097901 131612553863168 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:41:33.191482 131612553863168 traffic_director.py:259] Loaded NEG "psm-interop-server-20240408-2340-vugso-psm-grpc-server" in zone us-central1-a
I0408 23:41:33.191670 131612553863168 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240408-2340-vugso: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2340-vugso-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2340-vugso-psm-grpc-server', zone='us-central1-a')}
I0408 23:41:33.194662 131612553863168 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2340-vugso-psm-grpc-server
maxRatePerEndpoint: 5
...

I0408 23:41:34.311146 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619693200-6159e5872d4f7-ba38ff73-b9af51df
I0408 23:42:21.850380 131612553863168 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240408-2340-vugso to report all backends healthy: ['psm-interop-server-20240408-2340-vugso-psm-grpc-server']
I0408 23:42:26.354169 131612553863168 compute.py:494] Backend psm-interop-server-20240408-2340-vugso-psm-grpc-server in zone us-central1-a reported healthy
I0408 23:42:30.358730 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0408 23:42:30.358991 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0408 23:42:30.359115 131612553863168 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240408-2340-vugso: server_target=xds:///psm-grpc-server:19210 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0408 23:42:30.359335 131612553863168 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2340-vugso%22;timeRange=2024-04-08T23:42:30.359222Z%2F2024-04-09T00:12:30.359222Z?project=grpc-testing
I0408 23:42:30.360907 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240408-2340-vugso
labels:
name: psm-interop-client-20240408-2340-vugso
owner: xds-k8s-interop-test
...

I0408 23:42:30.371071 131612553863168 k8s_base_runner.py:367] Namespace psm-interop-client-20240408-2340-vugso created
I0408 23:42:30.371276 131612553863168 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2340-vugso/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:42:30.895189 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2340-vugso
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:42:30.932258 131612553863168 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0408 23:42:30.939689 131612553863168 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2340-vugso
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-7mpqylo2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-7mpqylo2
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-7mpqylo2
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:19210"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0408 23:42:30.980820 131612553863168 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0408 23:42:30.981039 131612553863168 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0408 23:42:31.017869 131612553863168 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6bdbb9577c-bxknm']
I0408 23:42:31.018106 131612553863168 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-6bdbb9577c-bxknm to start
I0408 23:42:53.297659 131612553863168 k8s_base_runner.py:985] Pod psm-grpc-client-6bdbb9577c-bxknm ready, IP: 10.120.17.85
I0408 23:42:53.297904 131612553863168 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-6bdbb9577c-bxknm to baseline_test/test_app_logs/psm-interop-client-20240408-2340-vugso_psm-grpc-client-6bdbb9577c-bxknm.log
I0408 23:42:53.298399 131612448900672 k8s_log_collector.py:71] [ns/psm-interop-client-20240408-2340-vugso] Starting log collection thread 131612448900672 for psm-grpc-client-6bdbb9577c-bxknm
I0408 23:42:53.298578 131612553863168 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0408 23:42:56.355310 131612553863168 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0408 23:42:56.355800 131612553863168 client_app.py:292] [psm-grpc-client-6bdbb9577c-bxknm] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0408 23:42:56.396636 131612553863168 client_app.py:320] [psm-grpc-client-6bdbb9577c-bxknm] xDS control plane channel:
I0408 23:43:06.420080 131612553863168 client_app.py:320] [psm-grpc-client-6bdbb9577c-bxknm] xDS control plane channel:
I0408 23:43:16.441863 131612553863168 client_app.py:320] [psm-grpc-client-6bdbb9577c-bxknm] xDS control plane channel:
I0408 23:43:26.463289 131612553863168 client_app.py:320] [psm-grpc-client-6bdbb9577c-bxknm] xDS control plane channel:
I0408 23:43:28.474248 131612553863168 client_app.py:331] [psm-grpc-client-6bdbb9577c-bxknm] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0408 23:43:28.474586 131612553863168 client_app.py:302] [psm-grpc-client-6bdbb9577c-bxknm] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0408 23:43:28.474855 131612553863168 client_app.py:252] [psm-grpc-client-6bdbb9577c-bxknm] Waiting to report a READY channel to xds:///psm-grpc-server:19210
I0408 23:43:28.479655 131612553863168 client_app.py:379] [psm-grpc-client-6bdbb9577c-bxknm] Server channel:
I0408 23:43:28.487685 131612553863168 client_app.py:392] [psm-grpc-client-6bdbb9577c-bxknm] Found subchannel in state READY:
I0408 23:43:28.488119 131612553863168 client_app.py:263] [psm-grpc-client-6bdbb9577c-bxknm] Channel to xds:///psm-grpc-server:19210 transitioned to state READY:
I0408 23:43:28.488269 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0408 23:43:28.488350 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.8_test_client_xds_config_exists ---
I0408 23:43:28.488548 131612553863168 grpc.py:75] [psm-grpc-client-6bdbb9577c-bxknm:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:43:28.501900 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.8_test_client_xds_config_exists ---
I0408 23:43:28.502089 131612553863168 xds_k8s_testcase.py:299] --- Starting subTest BaselineTest.test_traffic_director_grpc_setup.9_test_server_received_rpcs_from_test_client ---
I0408 23:43:28.502273 131612553863168 grpc.py:75] [psm-grpc-client-6bdbb9577c-bxknm:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:43:32.532735 131612553863168 xds_k8s_testcase.py:669] [psm-grpc-client-6bdbb9577c-bxknm] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f8456c994-sz8nq: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5f8456c994-sz8nq: 100

I0408 23:43:32.532982 131612553863168 xds_k8s_testcase.py:304] --- Finished subTest BaselineTest.test_traffic_director_grpc_setup.9_test_server_received_rpcs_from_test_client ---
I0408 23:43:32.533083 131612553863168 xds_k8s_testcase.py:755] ----- TestMethod BaselineTest.test_traffic_director_grpc_setup teardown -----
I0408 23:43:32.602433 131612553863168 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2340-vugso"
I0408 23:43:33.339151 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619812610-6159e5f90e2ee-12e1169a-4abf4019
I0408 23:43:46.393984 131612553863168 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240408-2340-vugso"
I0408 23:43:46.483733 131612553863168 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240408-2340-vugso"
I0408 23:43:46.894234 131612553863168 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2340-vugso"
I0408 23:43:47.575545 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619826901-6159e606af2ef-d1dea8fb-16b6a576
I0408 23:43:49.894746 131612553863168 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240408-2340-vugso"
I0408 23:43:50.180815 131612553863168 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2340-vugso"
I0408 23:43:51.097861 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619830189-6159e609d1d94-ba51f525-4148ed7f
I0408 23:43:53.419094 131612553863168 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240408-2340-vugso"
I0408 23:43:53.937241 131612553863168 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240408-2340-vugso"
I0408 23:43:54.712393 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619833942-6159e60d6613b-b8fc9972-abf0f7ec
I0408 23:44:27.074150 131612553863168 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240408-2340-vugso"
I0408 23:44:27.613061 131612553863168 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240408-2340-vugso"
I0408 23:44:28.143644 131612553863168 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240408-2340-vugso"
I0408 23:44:28.662456 131612553863168 compute.py:614] Waiting 600 sec for compute operation id: operation-1712619868217-6159e62e16191-13b0aa21-2d5b9034
I0408 23:44:33.119590 131612553863168 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0408 23:45:13.288012 131612553863168 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0408 23:45:13.288260 131612553863168 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2340-vugso/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:45:13.794160 131612553863168 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0408 23:45:13.853134 131612553863168 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0408 23:45:13.853337 131612553863168 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240408-2340-vugso
I0408 23:45:43.983598 131612553863168 k8s_base_runner.py:941] Namespace psm-interop-client-20240408-2340-vugso deleted
I0408 23:45:43.983869 131612553863168 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0408 23:45:54.080335 131612553863168 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0408 23:45:54.080600 131612553863168 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0408 23:45:54.142147 131612553863168 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0408 23:45:54.142359 131612553863168 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2340-vugso/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:45:54.518604 131612553863168 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0408 23:45:54.723837 131612553863168 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0408 23:45:54.724071 131612553863168 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2340-vugso
I0408 23:46:54.855561 131612553863168 k8s_base_runner.py:941] Namespace psm-interop-server-20240408-2340-vugso deleted
I0408 23:46:54.855892 131612553863168 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0408 23:46:54.856130 131612553863168 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-7mpqylo2:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2340-vugso%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-7mpqylo2%22;timeRange=2024-04-08T23:42:30.359216Z%2F2024-04-08T23:45:43.983835Z;cursorTimestamp=2024-04-08T23:42:56.355585Z?project=grpc-testing
I0408 23:46:54.856273 131612553863168 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-v849g28l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2340-vugso%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-v849g28l%22;timeRange=2024-04-08T23:41:23.280870Z%2F2024-04-08T23:46:54.855755Z;cursorTimestamp=2024-04-08T23:41:33.093051Z?project=grpc-testing
[ OK ] BaselineTest.test_traffic_director_grpc_setup
I0408 23:46:54.856500 131612553863168 base_testcase.py:67] ----- PSM Test Case PASSED: BaselineTest.test_traffic_director_grpc_setup -----

----------------------------------------------------------------------
Ran 1 test in 386.114s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/change_backend_service_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0408 23:46:55.922365 131361885954048 xds_k8s_testcase.py:162] ----- Testing ChangeBackendServiceTest -----
I0408 23:46:55.922470 131361885954048 xds_k8s_testcase.py:163] Logs timezone: UTC
I0408 23:46:55.922628 131361885954048 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0408 23:46:55.935332 131361885954048 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0408 23:46:55.948149 131361885954048 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0408 23:46:55.952043 131361885954048 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240408-2346-mxf4v
I0408 23:46:56.181566 131361885954048 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/change_backend_service_test
I0408 23:46:56.474550 131361885954048 xds_k8s_testcase.py:731] Found unused xds port: 24578
I0408 23:46:56.476766 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0408 23:46:56.477318 131361885954048 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240408-2346-mxf4v"
I0408 23:46:56.480416 131361885954048 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240408-2346-mxf4v
type: GRPC
...

I0408 23:46:57.240804 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620016547-6159e6bb8b86d-210350c2-e6a8645e
I0408 23:46:59.587603 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0408 23:46:59.587815 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0408 23:46:59.587916 131361885954048 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240408-2346-mxf4v"
I0408 23:46:59.593738 131361885954048 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2346-mxf4v
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240408-2346-mxf4v
protocol: GRPC
...

I0408 23:47:00.274136 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620019599-6159e6be74a53-542199a7-14a7f25e
I0408 23:47:15.672467 131361885954048 traffic_director.py:309] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20240408-2346-mxf4v"
I0408 23:47:15.676323 131361885954048 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2346-mxf4v
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20240408-2346-mxf4v
protocol: GRPC
...

I0408 23:47:16.331469 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620035683-6159e6cdcb306-2108863b-4b3fae8c
I0408 23:47:35.944547 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0408 23:47:35.944759 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0408 23:47:35.944869 131361885954048 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240408-2346-mxf4v": psm-grpc-server:24578 -> psm-interop-backend-service-20240408-2346-mxf4v
I0408 23:47:35.952193 131361885954048 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2346-mxf4v
hostRules:
- hosts:
- psm-grpc-server:24578
pathMatcher: psm-interop-path-matcher-20240408-2346-mxf4v
name: psm-interop-url-map-20240408-2346-mxf4v
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2346-mxf4v
name: psm-interop-path-matcher-20240408-2346-mxf4v
...

I0408 23:47:36.640479 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620055954-6159e6e12031c-95c85135-7d2ede79
I0408 23:47:38.951530 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0408 23:47:38.951714 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0408 23:47:38.951804 131361885954048 traffic_director.py:578] Creating target psm-interop-target-proxy-20240408-2346-mxf4v proxy "GRPC" to URL map psm-interop-url-map-20240408-2346-mxf4v
I0408 23:47:38.953200 131361885954048 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240408-2346-mxf4v
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240408-2346-mxf4v
validate_for_proxyless: true
...

I0408 23:47:39.503173 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620058959-6159e6e3fde6f-32439f21-6788f1a3
I0408 23:47:41.838323 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0408 23:47:41.838518 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0408 23:47:41.838620 131361885954048 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240408-2346-mxf4v" in network "default-vpc": 0.0.0.0:24578 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2346-mxf4v
I0408 23:47:41.840595 131361885954048 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240408-2346-mxf4v
network: global/networks/default-vpc
portRange: 24578
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2346-mxf4v
...

I0408 23:47:42.774957 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620061842-6159e6e6bdccd-9a02c47f-9519878c
I0408 23:47:57.995832 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0408 23:47:57.996077 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0408 23:47:57.996206 131361885954048 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240408-2346-mxf4v: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0408 23:47:57.996477 131361885954048 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2346-mxf4v%22;timeRange=2024-04-08T23:47:57.996319Z%2F2024-04-09T00:17:57.996319Z?project=grpc-testing
I0408 23:47:58.000436 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240408-2346-mxf4v
labels:
name: psm-interop-server-20240408-2346-mxf4v
owner: xds-k8s-interop-test
...

I0408 23:47:58.060667 131361885954048 k8s_base_runner.py:367] Namespace psm-interop-server-20240408-2346-mxf4v created
I0408 23:47:58.062864 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2346-mxf4v-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:47:58.103063 131361885954048 k8s_base_runner.py:367] Service psm-grpc-server created
I0408 23:47:58.103262 131361885954048 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240408-2346-mxf4v
I0408 23:47:58.143713 131361885954048 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240408-2346-mxf4v: neg_name=psm-interop-server-20240408-2346-mxf4v-psm-grpc-server, port=8080, zones=['us-central1-a']
I0408 23:47:58.143941 131361885954048 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2346-mxf4v/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:47:58.622248 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:47:58.651063 131361885954048 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0408 23:47:58.662441 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-orzgpwr6
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-orzgpwr6
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-orzgpwr6
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2346-mxf4v-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:47:58.702428 131361885954048 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0408 23:47:58.702602 131361885954048 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0408 23:47:58.750744 131361885954048 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6549867cc4-ngrpt']
I0408 23:47:58.750941 131361885954048 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-6549867cc4-ngrpt to start
I0408 23:48:02.866143 131361885954048 k8s_base_runner.py:985] Pod psm-grpc-server-6549867cc4-ngrpt ready, IP: 10.120.12.57
I0408 23:48:02.866363 131361885954048 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-6549867cc4-ngrpt to change_backend_service_test/test_app_logs/psm-interop-server-20240408-2346-mxf4v_psm-grpc-server-6549867cc4-ngrpt.log
I0408 23:48:02.866668 131361789482560 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2346-mxf4v] Starting log collection thread 131361789482560 for psm-grpc-server-6549867cc4-ngrpt
I0408 23:48:02.866751 131361885954048 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0408 23:48:07.985628 131361885954048 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0408 23:48:07.985891 131361885954048 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240408-2346-mxf4v: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0408 23:48:07.986147 131361885954048 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2346-mxf4v%22;timeRange=2024-04-08T23:48:07.986047Z%2F2024-04-09T00:18:07.986047Z?project=grpc-testing
I0408 23:48:07.986230 131361885954048 k8s_base_runner.py:377] Reusing namespace: psm-interop-server-20240408-2346-mxf4v
I0408 23:48:08.015498 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:48:08.035282 131361885954048 k8s_base_runner.py:367] Service psm-grpc-server-alt created
I0408 23:48:08.035518 131361885954048 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240408-2346-mxf4v
I0408 23:48:08.076700 131361885954048 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240408-2346-mxf4v: neg_name=psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0408 23:48:08.076901 131361885954048 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2346-mxf4v/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:48:08.581268 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:48:08.596639 131361885954048 k8s_base_runner.py:367] ServiceAccount psm-grpc-server-alt created
I0408 23:48:08.599000 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240408-2346-mxf4v
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-6cq97ukt
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-6cq97ukt
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-6cq97ukt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:48:08.622380 131361885954048 k8s_base_runner.py:367] Deployment psm-grpc-server-alt created
I0408 23:48:08.622556 131361885954048 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0408 23:48:09.665892 131361885954048 k8s_base_runner.py:972] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:48:09.666192 131361885954048 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-alt-7896c8f9d7-jpv25 to start
I0408 23:48:11.756126 131361885954048 k8s_base_runner.py:985] Pod psm-grpc-server-alt-7896c8f9d7-jpv25 ready, IP: 10.120.10.212
I0408 23:48:11.756390 131361885954048 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-alt-7896c8f9d7-jpv25 to change_backend_service_test/test_app_logs/psm-interop-server-20240408-2346-mxf4v_psm-grpc-server-alt-7896c8f9d7-jpv25.log
I0408 23:48:11.756945 131361708963392 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2346-mxf4v] Starting log collection thread 131361708963392 for psm-grpc-server-alt-7896c8f9d7-jpv25
I0408 23:48:11.757142 131361885954048 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0408 23:48:17.897814 131361885954048 k8s_base_runner.py:953] Deployment psm-grpc-server-alt has 1 replicas available
I0408 23:48:17.898175 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0408 23:48:17.898287 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0408 23:48:17.909375 131361885954048 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:48:17.986152 131361885954048 traffic_director.py:259] Loaded NEG "psm-interop-server-20240408-2346-mxf4v-psm-grpc-server" in zone us-central1-a
I0408 23:48:17.986339 131361885954048 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240408-2346-mxf4v: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2346-mxf4v-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2346-mxf4v-psm-grpc-server', zone='us-central1-a')}
I0408 23:48:17.989432 131361885954048 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2346-mxf4v-psm-grpc-server
maxRatePerEndpoint: 5
...

I0408 23:48:19.090551 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620097991-6159e70937332-57b7f0be-6614d284
I0408 23:49:00.101210 131361885954048 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240408-2346-mxf4v to report all backends healthy: ['psm-interop-server-20240408-2346-mxf4v-psm-grpc-server']
I0408 23:49:04.615162 131361885954048 compute.py:494] Backend psm-interop-server-20240408-2346-mxf4v-psm-grpc-server in zone us-central1-a reported healthy
I0408 23:49:09.377847 131361885954048 traffic_director.py:337] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:49:09.485851 131361885954048 traffic_director.py:340] Loaded NEG "psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt" in zone us-central1-a
I0408 23:49:09.486039 131361885954048 traffic_director.py:347] Adding backends to Backend Service psm-interop-backend-service-alt-20240408-2346-mxf4v: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt', zone='us-central1-a')}
I0408 23:49:09.489115 131361885954048 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2346-mxf4v-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0408 23:49:10.390253 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620149495-6159e73a55699-dccc6125-5b6c1988
I0408 23:49:59.903960 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0408 23:49:59.904160 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0408 23:49:59.904267 131361885954048 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240408-2346-mxf4v: server_target=xds:///psm-grpc-server:24578 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0408 23:49:59.904482 131361885954048 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2346-mxf4v%22;timeRange=2024-04-08T23:49:59.904362Z%2F2024-04-09T00:19:59.904362Z?project=grpc-testing
I0408 23:49:59.905918 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240408-2346-mxf4v
labels:
name: psm-interop-client-20240408-2346-mxf4v
owner: xds-k8s-interop-test
...

I0408 23:49:59.931620 131361885954048 k8s_base_runner.py:367] Namespace psm-interop-client-20240408-2346-mxf4v created
I0408 23:49:59.931723 131361885954048 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2346-mxf4v/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:50:00.403581 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2346-mxf4v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:50:00.433267 131361885954048 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0408 23:50:00.440703 131361885954048 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2346-mxf4v
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q43xd8xs
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q43xd8xs
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-q43xd8xs
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:24578"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0408 23:50:00.478139 131361885954048 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0408 23:50:00.478342 131361885954048 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0408 23:50:00.515120 131361885954048 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7b84f74b64-q8vbw']
I0408 23:50:00.515342 131361885954048 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-7b84f74b64-q8vbw to start
I0408 23:50:04.592913 131361885954048 k8s_base_runner.py:985] Pod psm-grpc-client-7b84f74b64-q8vbw ready, IP: 10.120.17.86
I0408 23:50:04.593134 131361885954048 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-7b84f74b64-q8vbw to change_backend_service_test/test_app_logs/psm-interop-client-20240408-2346-mxf4v_psm-grpc-client-7b84f74b64-q8vbw.log
I0408 23:50:04.593621 131361700570688 k8s_log_collector.py:71] [ns/psm-interop-client-20240408-2346-mxf4v] Starting log collection thread 131361700570688 for psm-grpc-client-7b84f74b64-q8vbw
I0408 23:50:04.593791 131361885954048 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0408 23:50:07.664179 131361885954048 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0408 23:50:07.664584 131361885954048 client_app.py:292] [psm-grpc-client-7b84f74b64-q8vbw] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0408 23:50:07.705450 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:50:17.729737 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:50:27.752406 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:50:37.775248 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:50:47.799055 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:51:03.828604 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:51:28.864410 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:51:53.888532 131361885954048 client_app.py:320] [psm-grpc-client-7b84f74b64-q8vbw] xDS control plane channel:
I0408 23:51:55.898560 131361885954048 client_app.py:331] [psm-grpc-client-7b84f74b64-q8vbw] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0408 23:51:55.898742 131361885954048 client_app.py:302] [psm-grpc-client-7b84f74b64-q8vbw] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0408 23:51:55.898953 131361885954048 client_app.py:252] [psm-grpc-client-7b84f74b64-q8vbw] Waiting to report a READY channel to xds:///psm-grpc-server:24578
I0408 23:51:55.903351 131361885954048 client_app.py:379] [psm-grpc-client-7b84f74b64-q8vbw] Server channel:
I0408 23:51:55.910658 131361885954048 client_app.py:392] [psm-grpc-client-7b84f74b64-q8vbw] Found subchannel in state READY:
I0408 23:51:55.910820 131361885954048 client_app.py:263] [psm-grpc-client-7b84f74b64-q8vbw] Channel to xds:///psm-grpc-server:24578 transitioned to state READY:
I0408 23:51:55.910909 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0408 23:51:55.910960 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0408 23:51:55.911091 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0408 23:51:55.926357 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0408 23:51:55.926454 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0408 23:51:55.926592 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:51:59.997779 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6549867cc4-ngrpt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6549867cc4-ngrpt: 100

I0408 23:51:59.998036 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0408 23:51:59.998235 131361885954048 xds_k8s_testcase.py:299] --- Starting subTest ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0408 23:51:59.998329 131361885954048 traffic_director.py:499] Patching URL map "psm-interop-url-map-20240408-2346-mxf4v": psm-grpc-server:24578 -> psm-interop-backend-service-alt-20240408-2346-mxf4v
I0408 23:52:00.001785 131361885954048 compute.py:544] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240408-2346-mxf4v
hostRules:
- hosts:
- psm-grpc-server:24578
pathMatcher: psm-interop-path-matcher-20240408-2346-mxf4v
name: psm-interop-url-map-20240408-2346-mxf4v
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240408-2346-mxf4v
name: psm-interop-path-matcher-20240408-2346-mxf4v
...

I0408 23:52:00.936187 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620320008-6159e7dcf2a81-661c2b05-eb3ee393
I0408 23:52:41.804177 131361885954048 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:52:41.804416 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:52:45.823330 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6549867cc4-ngrpt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6549867cc4-ngrpt: 100

I0408 23:52:45.823961 131361885954048 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7896c8f9d7-jpv25' not found in {'psm-grpc-server-6549867cc4-ngrpt': 100} : Server psm-grpc-server-alt-7896c8f9d7-jpv25 did not receive RPCs
I0408 23:52:46.825213 131361885954048 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:52:46.825506 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:52:50.868499 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6549867cc4-ngrpt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6549867cc4-ngrpt: 100

I0408 23:52:50.868907 131361885954048 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7896c8f9d7-jpv25' not found in {'psm-grpc-server-6549867cc4-ngrpt': 100} : Server psm-grpc-server-alt-7896c8f9d7-jpv25 did not receive RPCs
I0408 23:52:51.870117 131361885954048 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:52:51.870392 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:52:55.866145 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6549867cc4-ngrpt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6549867cc4-ngrpt: 100

I0408 23:52:55.866543 131361885954048 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7896c8f9d7-jpv25' not found in {'psm-grpc-server-6549867cc4-ngrpt': 100} : Server psm-grpc-server-alt-7896c8f9d7-jpv25 did not receive RPCs
I0408 23:52:56.867515 131361885954048 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:52:56.867793 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:53:03.340427 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7896c8f9d7-jpv25: 72
psm-grpc-server-6549867cc4-ngrpt: 28
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7896c8f9d7-jpv25: 72
psm-grpc-server-6549867cc4-ngrpt: 28

I0408 23:53:03.340806 131361885954048 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6549867cc4-ngrpt' not found in ['psm-grpc-server-alt-7896c8f9d7-jpv25'] : Unexpected server psm-grpc-server-6549867cc4-ngrpt received RPCs
I0408 23:53:04.342131 131361885954048 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-7896c8f9d7-jpv25']
I0408 23:53:04.342435 131361885954048 grpc.py:75] [psm-grpc-client-7b84f74b64-q8vbw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0408 23:53:08.367499 131361885954048 xds_k8s_testcase.py:669] [psm-grpc-client-7b84f74b64-q8vbw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7896c8f9d7-jpv25: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7896c8f9d7-jpv25: 100

I0408 23:53:08.367916 131361885954048 xds_k8s_testcase.py:304] --- Finished subTest ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0408 23:53:08.368010 131361885954048 xds_k8s_testcase.py:755] ----- TestMethod ChangeBackendServiceTest.test_change_backend_service teardown -----
I0408 23:53:08.442558 131361885954048 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2346-mxf4v"
I0408 23:53:09.224114 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620388446-6159e81e370f4-3da6d6cd-6b68aa8e
I0408 23:53:22.253649 131361885954048 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240408-2346-mxf4v"
I0408 23:53:22.336926 131361885954048 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240408-2346-mxf4v"
I0408 23:53:22.848441 131361885954048 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2346-mxf4v"
I0408 23:53:23.420892 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620402855-6159e82bf4e4c-4abe5892-1b28ac7c
I0408 23:53:25.723538 131361885954048 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240408-2346-mxf4v"
I0408 23:53:26.059109 131361885954048 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2346-mxf4v"
I0408 23:53:26.619815 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620406063-6159e82f0418b-ec03a52e-35034252
I0408 23:53:28.910934 131361885954048 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240408-2346-mxf4v"
I0408 23:53:29.254926 131361885954048 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240408-2346-mxf4v"
I0408 23:53:29.941822 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620409259-6159e8321051f-54f9b63e-afacbc1c
I0408 23:53:55.950508 131361885954048 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240408-2346-mxf4v"
I0408 23:53:56.637210 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620435954-6159e84b85c24-34964805-6a5f6d18
I0408 23:54:18.281886 131361885954048 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240408-2346-mxf4v"
I0408 23:54:18.744829 131361885954048 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240408-2346-mxf4v"
I0408 23:54:19.281702 131361885954048 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620458831-6159e86156eb7-2c94188d-d79b1791
I0408 23:54:23.758585 131361885954048 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0408 23:55:03.904356 131361885954048 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0408 23:55:03.904586 131361885954048 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2346-mxf4v/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:55:04.316682 131361885954048 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0408 23:55:04.373593 131361885954048 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0408 23:55:04.373787 131361885954048 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240408-2346-mxf4v
I0408 23:55:34.489603 131361885954048 k8s_base_runner.py:941] Namespace psm-interop-client-20240408-2346-mxf4v deleted
I0408 23:55:34.489862 131361885954048 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0408 23:55:44.565715 131361885954048 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0408 23:55:44.565980 131361885954048 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0408 23:55:45.618964 131361885954048 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0408 23:55:45.619156 131361885954048 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2346-mxf4v/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:55:46.138099 131361885954048 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0408 23:55:46.187610 131361885954048 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0408 23:55:46.187817 131361885954048 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2346-mxf4v
I0408 23:57:16.350149 131361885954048 k8s_base_runner.py:941] Namespace psm-interop-server-20240408-2346-mxf4v deleted
I0408 23:57:16.350364 131361885954048 k8s_base_runner.py:873] Deleting deployment psm-grpc-server-alt
I0408 23:57:16.372210 131361885954048 k8s_base_runner.py:891] Deleting service psm-grpc-server-alt
I0408 23:57:16.397952 131361885954048 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2346-mxf4v/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:57:16.822813 131361885954048 k8s_base_runner.py:907] Deleting service account psm-grpc-server-alt
I0408 23:57:16.845599 131361885954048 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2346-mxf4v
I0408 23:57:16.850323 131361885954048 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0408 23:57:16.850556 131361885954048 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-q43xd8xs:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2346-mxf4v%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-q43xd8xs%22;timeRange=2024-04-08T23:49:59.904355Z%2F2024-04-08T23:55:34.489836Z;cursorTimestamp=2024-04-08T23:50:07.664410Z?project=grpc-testing
I0408 23:57:16.850691 131361885954048 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-orzgpwr6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2346-mxf4v%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-orzgpwr6%22;timeRange=2024-04-08T23:47:57.996310Z%2F2024-04-08T23:57:16.350340Z;cursorTimestamp=2024-04-08T23:48:07.985841Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
I0408 23:57:16.850934 131361885954048 base_testcase.py:67] ----- PSM Test Case PASSED: ChangeBackendServiceTest.test_change_backend_service -----

----------------------------------------------------------------------
Ran 1 test in 620.931s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0408 23:57:17.823566 126478893432832 xds_k8s_testcase.py:162] ----- Testing CustomLbTest -----
I0408 23:57:17.823835 126478893432832 xds_k8s_testcase.py:163] Logs timezone: UTC
I0408 23:57:17.824306 126478893432832 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0408 23:57:17.837692 126478893432832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0408 23:57:17.849774 126478893432832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0408 23:57:17.853189 126478893432832 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240408-2357-i38rj
I0408 23:57:18.128259 126478893432832 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/custom_lb_test
I0408 23:57:18.481639 126478893432832 xds_k8s_testcase.py:731] Found unused xds port: 35502
I0408 23:57:18.481905 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0408 23:57:18.481977 126478893432832 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240408-2357-i38rj"
I0408 23:57:18.484311 126478893432832 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240408-2357-i38rj
type: GRPC
...

I0408 23:57:19.212171 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620638554-6159e90cbc7c0-3bbef482-699c4e84
I0408 23:57:21.501318 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0408 23:57:21.501511 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0408 23:57:21.501596 126478893432832 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240408-2357-i38rj"
I0408 23:57:21.507361 126478893432832 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240408-2357-i38rj
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240408-2357-i38rj
protocol: GRPC
...

I0408 23:57:22.287495 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620641510-6159e90f8e404-1da9023b-fee40cc5
I0408 23:57:39.777395 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0408 23:57:39.777584 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0408 23:57:39.777672 126478893432832 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240408-2357-i38rj": psm-grpc-server:35502 -> psm-interop-backend-service-20240408-2357-i38rj
I0408 23:57:39.784946 126478893432832 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2357-i38rj
hostRules:
- hosts:
- psm-grpc-server:35502
pathMatcher: psm-interop-path-matcher-20240408-2357-i38rj
name: psm-interop-url-map-20240408-2357-i38rj
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240408-2357-i38rj
name: psm-interop-path-matcher-20240408-2357-i38rj
...

I0408 23:57:40.415850 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620659791-6159e920fd4c4-485d3053-df7b3ad8
I0408 23:57:42.722580 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0408 23:57:42.722756 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0408 23:57:42.722852 126478893432832 traffic_director.py:578] Creating target psm-interop-target-proxy-20240408-2357-i38rj proxy "GRPC" to URL map psm-interop-url-map-20240408-2357-i38rj
I0408 23:57:42.724336 126478893432832 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240408-2357-i38rj
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240408-2357-i38rj
validate_for_proxyless: true
...

I0408 23:57:43.231399 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620662726-6159e923c9edc-f80c3d81-9c799aa6
I0408 23:57:45.524719 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0408 23:57:45.524921 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0408 23:57:45.525021 126478893432832 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240408-2357-i38rj" in network "default-vpc": 0.0.0.0:35502 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2357-i38rj
I0408 23:57:45.526843 126478893432832 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240408-2357-i38rj
network: global/networks/default-vpc
portRange: 35502
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240408-2357-i38rj
...

I0408 23:57:46.239762 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620665529-6159e926762cc-3399be18-a5629900
I0408 23:58:07.917067 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0408 23:58:07.917271 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0408 23:58:07.917371 126478893432832 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240408-2357-i38rj: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0408 23:58:07.917587 126478893432832 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2357-i38rj%22;timeRange=2024-04-08T23:58:07.917464Z%2F2024-04-09T00:28:07.917464Z?project=grpc-testing
I0408 23:58:07.921259 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240408-2357-i38rj
labels:
name: psm-interop-server-20240408-2357-i38rj
owner: xds-k8s-interop-test
...

I0408 23:58:07.969607 126478893432832 k8s_base_runner.py:367] Namespace psm-interop-server-20240408-2357-i38rj created
I0408 23:58:07.971511 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2357-i38rj
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240408-2357-i38rj-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0408 23:58:08.004506 126478893432832 k8s_base_runner.py:367] Service psm-grpc-server created
I0408 23:58:08.004616 126478893432832 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240408-2357-i38rj
I0408 23:58:08.034832 126478893432832 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240408-2357-i38rj: neg_name=psm-interop-server-20240408-2357-i38rj-psm-grpc-server, port=8080, zones=['us-central1-a']
I0408 23:58:08.035030 126478893432832 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2357-i38rj/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:58:08.510240 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2357-i38rj
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:58:08.539542 126478893432832 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0408 23:58:08.550482 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240408-2357-i38rj
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lkz87bcx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lkz87bcx
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lkz87bcx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240408-2357-i38rj-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0408 23:58:08.587269 126478893432832 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0408 23:58:08.587526 126478893432832 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0408 23:58:08.628100 126478893432832 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-74c47bdb67-rtqtv']
I0408 23:58:08.628309 126478893432832 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-74c47bdb67-rtqtv to start
I0408 23:58:11.694367 126478893432832 k8s_base_runner.py:985] Pod psm-grpc-server-74c47bdb67-rtqtv ready, IP: 10.120.9.40
I0408 23:58:11.694614 126478893432832 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-74c47bdb67-rtqtv to custom_lb_test/test_app_logs/psm-interop-server-20240408-2357-i38rj_psm-grpc-server-74c47bdb67-rtqtv.log
I0408 23:58:11.695140 126478796969536 k8s_log_collector.py:71] [ns/psm-interop-server-20240408-2357-i38rj] Starting log collection thread 126478796969536 for psm-grpc-server-74c47bdb67-rtqtv
I0408 23:58:11.695315 126478893432832 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0408 23:58:17.781490 126478893432832 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0408 23:58:17.781753 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0408 23:58:17.781857 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0408 23:58:17.786658 126478893432832 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0408 23:58:17.878207 126478893432832 traffic_director.py:259] Loaded NEG "psm-interop-server-20240408-2357-i38rj-psm-grpc-server" in zone us-central1-a
I0408 23:58:17.878408 126478893432832 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240408-2357-i38rj: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240408-2357-i38rj-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2357-i38rj-psm-grpc-server', zone='us-central1-a')}
I0408 23:58:17.881411 126478893432832 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240408-2357-i38rj-psm-grpc-server
maxRatePerEndpoint: 5
...

I0408 23:58:18.933731 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620697888-6159e94552658-29093ceb-2c58785f
I0408 23:59:10.739723 126478893432832 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240408-2357-i38rj to report all backends healthy: ['psm-interop-server-20240408-2357-i38rj-psm-grpc-server']
I0408 23:59:11.178497 126478893432832 compute.py:494] Backend psm-interop-server-20240408-2357-i38rj-psm-grpc-server in zone us-central1-a reported healthy
I0408 23:59:15.183256 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0408 23:59:15.183524 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0408 23:59:15.183638 126478893432832 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240408-2357-i38rj: server_target=xds:///psm-grpc-server:35502 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0408 23:59:15.183869 126478893432832 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2357-i38rj%22;timeRange=2024-04-08T23:59:15.183740Z%2F2024-04-09T00:29:15.183740Z?project=grpc-testing
I0408 23:59:15.185451 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240408-2357-i38rj
labels:
name: psm-interop-client-20240408-2357-i38rj
owner: xds-k8s-interop-test
...

I0408 23:59:15.195261 126478893432832 k8s_base_runner.py:367] Namespace psm-interop-client-20240408-2357-i38rj created
I0408 23:59:15.195363 126478893432832 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2357-i38rj/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0408 23:59:15.637721 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2357-i38rj
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0408 23:59:15.664962 126478893432832 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0408 23:59:15.671788 126478893432832 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240408-2357-i38rj
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ncr91v63
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ncr91v63
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ncr91v63
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:35502"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0408 23:59:15.709550 126478893432832 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0408 23:59:15.709754 126478893432832 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0408 23:59:15.749273 126478893432832 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-84fdc759c4-55rtz']
I0408 23:59:15.749508 126478893432832 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-84fdc759c4-55rtz to start
I0408 23:59:18.811015 126478893432832 k8s_base_runner.py:985] Pod psm-grpc-client-84fdc759c4-55rtz ready, IP: 10.120.9.41
I0408 23:59:18.811223 126478893432832 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-84fdc759c4-55rtz to custom_lb_test/test_app_logs/psm-interop-client-20240408-2357-i38rj_psm-grpc-client-84fdc759c4-55rtz.log
I0408 23:59:18.811689 126478788576832 k8s_log_collector.py:71] [ns/psm-interop-client-20240408-2357-i38rj] Starting log collection thread 126478788576832 for psm-grpc-client-84fdc759c4-55rtz
I0408 23:59:18.811867 126478893432832 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0408 23:59:22.889795 126478893432832 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0408 23:59:22.890181 126478893432832 client_app.py:292] [psm-grpc-client-84fdc759c4-55rtz] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0408 23:59:22.933925 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0408 23:59:32.955142 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0408 23:59:42.976529 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0408 23:59:52.997544 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0409 00:00:03.023971 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0409 00:00:19.051733 126478893432832 client_app.py:320] [psm-grpc-client-84fdc759c4-55rtz] xDS control plane channel:
I0409 00:00:21.061199 126478893432832 client_app.py:331] [psm-grpc-client-84fdc759c4-55rtz] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0409 00:00:21.061464 126478893432832 client_app.py:302] [psm-grpc-client-84fdc759c4-55rtz] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0409 00:00:21.061702 126478893432832 client_app.py:252] [psm-grpc-client-84fdc759c4-55rtz] Waiting to report a READY channel to xds:///psm-grpc-server:35502
I0409 00:00:21.066156 126478893432832 client_app.py:379] [psm-grpc-client-84fdc759c4-55rtz] Server channel:
I0409 00:00:21.075446 126478893432832 client_app.py:392] [psm-grpc-client-84fdc759c4-55rtz] Found subchannel in state READY:
I0409 00:00:21.075760 126478893432832 client_app.py:263] [psm-grpc-client-84fdc759c4-55rtz] Channel to xds:///psm-grpc-server:35502 transitioned to state READY:
I0409 00:00:21.075935 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0409 00:00:21.076021 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0409 00:00:21.076211 126478893432832 grpc.py:75] [psm-grpc-client-84fdc759c4-55rtz:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0409 00:00:21.090789 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0409 00:00:21.090986 126478893432832 xds_k8s_testcase.py:299] --- Starting subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0409 00:00:21.091196 126478893432832 grpc.py:75] [psm-grpc-client-84fdc759c4-55rtz:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0409 00:00:31.099628 126478893432832 grpc.py:75] [psm-grpc-client-84fdc759c4-55rtz:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0409 00:00:31.107739 126478893432832 xds_k8s_testcase.py:430] [psm-grpc-client-84fdc759c4-55rtz] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 249
result:
(15, DATA_LOSS): 248

I0409 00:00:31.108370 126478893432832 xds_k8s_testcase.py:304] --- Finished subTest CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0409 00:00:31.108474 126478893432832 xds_k8s_testcase.py:755] ----- TestMethod CustomLbTest.test_custom_lb_config teardown -----
I0409 00:00:31.172113 126478893432832 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240408-2357-i38rj"
I0409 00:00:31.915225 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620831175-6159e9c46f4fb-d89d713b-5b47d5f0
I0409 00:01:00.261196 126478893432832 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240408-2357-i38rj"
I0409 00:01:00.385118 126478893432832 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240408-2357-i38rj"
I0409 00:01:00.827695 126478893432832 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240408-2357-i38rj"
I0409 00:01:01.347175 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620860834-6159e9e0b8461-3c810688-b49a3bf4
I0409 00:01:03.660444 126478893432832 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240408-2357-i38rj"
I0409 00:01:04.035142 126478893432832 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240408-2357-i38rj"
I0409 00:01:04.647070 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620864039-6159e9e3c6c24-baead909-ae91ab02
I0409 00:01:06.955891 126478893432832 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240408-2357-i38rj"
I0409 00:01:07.355224 126478893432832 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240408-2357-i38rj"
I0409 00:01:08.019965 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620867360-6159e9e6f15b0-ae4531d1-1ad47b7d
I0409 00:01:36.253694 126478893432832 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240408-2357-i38rj"
I0409 00:01:36.699913 126478893432832 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240408-2357-i38rj"
I0409 00:01:37.084700 126478893432832 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240408-2357-i38rj"
I0409 00:01:37.573709 126478893432832 compute.py:614] Waiting 600 sec for compute operation id: operation-1712620897160-6159ea035cd9e-3346e623-c8d31aca
I0409 00:01:41.966943 126478893432832 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0409 00:02:22.100816 126478893432832 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0409 00:02:22.101101 126478893432832 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240408-2357-i38rj/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:02:22.543643 126478893432832 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0409 00:02:22.591266 126478893432832 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0409 00:02:22.591485 126478893432832 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240408-2357-i38rj
I0409 00:02:52.705924 126478893432832 k8s_base_runner.py:941] Namespace psm-interop-client-20240408-2357-i38rj deleted
I0409 00:02:52.706149 126478893432832 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0409 00:03:02.781368 126478893432832 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0409 00:03:02.781587 126478893432832 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0409 00:03:03.831261 126478893432832 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0409 00:03:03.831494 126478893432832 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240408-2357-i38rj/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:03:04.327921 126478893432832 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0409 00:03:04.366442 126478893432832 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0409 00:03:04.366645 126478893432832 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240408-2357-i38rj
I0409 00:03:34.445205 126478893432832 k8s_base_runner.py:941] Namespace psm-interop-server-20240408-2357-i38rj deleted
I0409 00:03:34.445540 126478893432832 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0409 00:03:34.445754 126478893432832 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-ncr91v63:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240408-2357-i38rj%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-ncr91v63%22;timeRange=2024-04-08T23:59:15.183733Z%2F2024-04-09T00:02:52.706122Z;cursorTimestamp=2024-04-08T23:59:22.890016Z?project=grpc-testing
I0409 00:03:34.445916 126478893432832 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-lkz87bcx:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240408-2357-i38rj%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lkz87bcx%22;timeRange=2024-04-08T23:58:07.917458Z%2F2024-04-09T00:03:34.445407Z;cursorTimestamp=2024-04-08T23:58:17.781699Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0409 00:03:34.446140 126478893432832 base_testcase.py:67] ----- PSM Test Case PASSED: CustomLbTest.test_custom_lb_config -----

----------------------------------------------------------------------
Ran 1 test in 376.625s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/failover_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0409 00:03:35.451218 128371071918080 xds_k8s_testcase.py:162] ----- Testing FailoverTest -----
I0409 00:03:35.451329 128371071918080 xds_k8s_testcase.py:163] Logs timezone: UTC
I0409 00:03:35.451623 128371071918080 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0409 00:03:35.463923 128371071918080 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0409 00:03:35.476181 128371071918080 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0409 00:03:35.479814 128371071918080 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240409-0003-aex3b
I0409 00:03:35.826497 128371071918080 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/failover_test
I0409 00:03:36.089209 128371071918080 xds_k8s_testcase.py:731] Found unused xds port: 64474
I0409 00:03:36.090879 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.00_create_health_check ---
I0409 00:03:36.090972 128371071918080 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240409-0003-aex3b"
I0409 00:03:36.093106 128371071918080 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240409-0003-aex3b
type: GRPC
...

I0409 00:03:36.802813 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621016148-6159ea74d699f-a3b6eda3-d8fbf9bd
I0409 00:03:39.128764 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.00_create_health_check ---
I0409 00:03:39.128930 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.01_create_backend_services ---
I0409 00:03:39.129004 128371071918080 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240409-0003-aex3b"
I0409 00:03:39.134687 128371071918080 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240409-0003-aex3b
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240409-0003-aex3b
protocol: GRPC
...

I0409 00:03:40.063703 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621019136-6159ea77b0399-8a62698e-e38a6fb6
I0409 00:04:14.685061 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.01_create_backend_services ---
I0409 00:04:14.685275 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.02_create_url_map ---
I0409 00:04:14.685387 128371071918080 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240409-0003-aex3b": psm-grpc-server:64474 -> psm-interop-backend-service-20240409-0003-aex3b
I0409 00:04:14.692898 128371071918080 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0003-aex3b
hostRules:
- hosts:
- psm-grpc-server:64474
pathMatcher: psm-interop-path-matcher-20240409-0003-aex3b
name: psm-interop-url-map-20240409-0003-aex3b
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0003-aex3b
name: psm-interop-path-matcher-20240409-0003-aex3b
...

I0409 00:04:15.302244 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621054695-6159ea9999732-72b616bc-d22fb8c5
I0409 00:04:17.563816 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.02_create_url_map ---
I0409 00:04:17.563954 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.03_create_target_proxy ---
I0409 00:04:17.564015 128371071918080 traffic_director.py:578] Creating target psm-interop-target-proxy-20240409-0003-aex3b proxy "GRPC" to URL map psm-interop-url-map-20240409-0003-aex3b
I0409 00:04:17.565382 128371071918080 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240409-0003-aex3b
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240409-0003-aex3b
validate_for_proxyless: true
...

I0409 00:04:18.104677 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621057571-6159ea9c57b03-0b1c41fa-6d2b6dae
I0409 00:04:20.388363 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.03_create_target_proxy ---
I0409 00:04:20.388573 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.04_create_forwarding_rule ---
I0409 00:04:20.388682 128371071918080 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240409-0003-aex3b" in network "default-vpc": 0.0.0.0:64474 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0003-aex3b
I0409 00:04:20.390573 128371071918080 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240409-0003-aex3b
network: global/networks/default-vpc
portRange: 64474
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0003-aex3b
...

I0409 00:04:21.370787 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621060392-6159ea9f08669-f1510b84-3aea16fe
I0409 00:05:08.939718 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.04_create_forwarding_rule ---
I0409 00:05:08.939928 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.05_start_test_servers ---
I0409 00:05:08.940028 128371071918080 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240409-0003-aex3b: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0409 00:05:08.940235 128371071918080 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0003-aex3b%22;timeRange=2024-04-09T00:05:08.940120Z%2F2024-04-09T00:35:08.940120Z?project=grpc-testing
I0409 00:05:08.943757 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240409-0003-aex3b
labels:
name: psm-interop-server-20240409-0003-aex3b
owner: xds-k8s-interop-test
...

I0409 00:05:08.989772 128371071918080 k8s_base_runner.py:367] Namespace psm-interop-server-20240409-0003-aex3b created
I0409 00:05:08.991923 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0003-aex3b
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0003-aex3b-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:05:09.023855 128371071918080 k8s_base_runner.py:367] Service psm-grpc-server created
I0409 00:05:09.024020 128371071918080 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240409-0003-aex3b
I0409 00:05:09.058229 128371071918080 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240409-0003-aex3b: neg_name=psm-interop-server-20240409-0003-aex3b-psm-grpc-server, port=8080, zones=['us-central1-a']
I0409 00:05:09.058388 128371071918080 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0003-aex3b/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:05:09.601338 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0003-aex3b
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:05:09.624796 128371071918080 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0409 00:05:09.635987 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0003-aex3b
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4ni2idyy
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4ni2idyy
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4ni2idyy
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0003-aex3b-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:05:09.668072 128371071918080 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0409 00:05:09.668218 128371071918080 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0409 00:05:10.733904 128371071918080 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-6c9c6b757b-4vqbl', 'psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:05:10.734213 128371071918080 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-6c9c6b757b-4vqbl to start
I0409 00:05:13.800156 128371071918080 k8s_base_runner.py:985] Pod psm-grpc-server-6c9c6b757b-4vqbl ready, IP: 10.120.10.215
I0409 00:05:13.800362 128371071918080 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-6c9c6b757b-4vqbl to failover_test/test_app_logs/psm-interop-server-20240409-0003-aex3b_psm-grpc-server-6c9c6b757b-4vqbl.log
I0409 00:05:13.800852 128370975430208 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0003-aex3b] Starting log collection thread 128370975430208 for psm-grpc-server-6c9c6b757b-4vqbl
I0409 00:05:13.801022 128371071918080 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-6c9c6b757b-gzjqd to start
I0409 00:05:13.833502 128371071918080 k8s_base_runner.py:985] Pod psm-grpc-server-6c9c6b757b-gzjqd ready, IP: 10.120.9.43
I0409 00:05:13.833648 128371071918080 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-6c9c6b757b-gzjqd to failover_test/test_app_logs/psm-interop-server-20240409-0003-aex3b_psm-grpc-server-6c9c6b757b-gzjqd.log
I0409 00:05:13.833986 128370967037504 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0003-aex3b] Starting log collection thread 128370967037504 for psm-grpc-server-6c9c6b757b-gzjqd
I0409 00:05:13.834134 128371071918080 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-6c9c6b757b-xzzk6 to start
I0409 00:05:13.874660 128371071918080 k8s_base_runner.py:985] Pod psm-grpc-server-6c9c6b757b-xzzk6 ready, IP: 10.120.17.89
I0409 00:05:13.874878 128371071918080 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-6c9c6b757b-xzzk6 to failover_test/test_app_logs/psm-interop-server-20240409-0003-aex3b_psm-grpc-server-6c9c6b757b-xzzk6.log
I0409 00:05:13.875295 128370958644800 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0003-aex3b] Starting log collection thread 128370958644800 for psm-grpc-server-6c9c6b757b-xzzk6
I0409 00:05:13.875491 128371071918080 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0409 00:05:20.984757 128371071918080 k8s_base_runner.py:953] Deployment psm-grpc-server has 3 replicas available
I0409 00:05:20.985009 128371071918080 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240409-0003-aex3b: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0409 00:05:20.985200 128371071918080 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0003-aex3b%22;timeRange=2024-04-09T00:05:20.985110Z%2F2024-04-09T00:35:20.985110Z?project=grpc-testing
I0409 00:05:20.985657 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240409-0003-aex3b
labels:
name: psm-interop-server-20240409-0003-aex3b
owner: xds-k8s-interop-test
...

I0409 00:05:21.144576 128371071918080 k8s_base_runner.py:367] Namespace psm-interop-server-20240409-0003-aex3b created
I0409 00:05:21.145429 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0003-aex3b
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:05:21.214335 128371071918080 k8s_base_runner.py:367] Service psm-grpc-server-alt created
I0409 00:05:21.214532 128371071918080 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240409-0003-aex3b
I0409 00:05:21.314718 128371071918080 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240409-0003-aex3b: neg_name=psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt, port=8080, zones=['us-west1-b']
I0409 00:05:21.314905 128371071918080 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0003-aex3b/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:05:21.950852 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0003-aex3b
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:05:22.011111 128371071918080 k8s_base_runner.py:367] ServiceAccount psm-grpc-server-alt created
I0409 00:05:22.013347 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0003-aex3b
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-v992apu2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-v992apu2
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-v992apu2
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:05:22.086515 128371071918080 k8s_base_runner.py:367] Deployment psm-grpc-server-alt created
I0409 00:05:22.086666 128371071918080 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0409 00:05:22.202240 128371071918080 k8s_base_runner.py:972] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:05:22.202424 128371071918080 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-alt-79dbc67778-hmssx to start
I0409 00:05:30.689363 128371071918080 k8s_base_runner.py:985] Pod psm-grpc-server-alt-79dbc67778-hmssx ready, IP: 10.108.10.135
I0409 00:05:30.689650 128371071918080 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-alt-79dbc67778-hmssx to failover_test/test_app_logs/psm-interop-server-20240409-0003-aex3b_psm-grpc-server-alt-79dbc67778-hmssx.log
I0409 00:05:30.690128 128370949203520 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0003-aex3b] Starting log collection thread 128370949203520 for psm-grpc-server-alt-79dbc67778-hmssx
I0409 00:05:30.690295 128371071918080 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0409 00:05:33.932918 128371071918080 k8s_base_runner.py:953] Deployment psm-grpc-server-alt has 1 replicas available
I0409 00:05:33.933178 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.05_start_test_servers ---
I0409 00:05:33.933263 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0409 00:05:33.939550 128371071918080 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:05:34.018478 128371071918080 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0003-aex3b-psm-grpc-server" in zone us-central1-a
I0409 00:05:34.018638 128371071918080 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0003-aex3b: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0003-aex3b-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server', zone='us-central1-a')}
I0409 00:05:34.021644 128371071918080 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server
maxRatePerEndpoint: 100
...

I0409 00:05:35.018095 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621134023-6159eae540b99-0f9360d1-d73d3fea
I0409 00:06:20.490356 128371071918080 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0003-aex3b to report all backends healthy: ['psm-interop-server-20240409-0003-aex3b-psm-grpc-server']
I0409 00:06:33.348094 128371071918080 compute.py:494] Backend psm-interop-server-20240409-0003-aex3b-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:06:37.395003 128371071918080 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:06:37.512739 128371071918080 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt" in zone us-west1-b
I0409 00:06:37.512947 128371071918080 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0003-aex3b: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0003-aex3b-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server', zone='us-central1-a')}
I0409 00:06:37.516479 128371071918080 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0003-aex3b-psm-grpc-server
maxRatePerEndpoint: 100
...

I0409 00:06:38.949038 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621197522-6159eb21cf659-7991c9de-49a87d18
I0409 00:07:11.627908 128371071918080 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0003-aex3b to report all backends healthy: ['psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt', 'psm-interop-server-20240409-0003-aex3b-psm-grpc-server']
I0409 00:07:13.135200 128371071918080 compute.py:494] Backend psm-interop-server-20240409-0003-aex3b-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:07:31.592563 128371071918080 compute.py:494] Backend psm-interop-server-20240409-0003-aex3b-psm-grpc-server-alt in zone us-west1-b reported healthy
I0409 00:07:35.597166 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0409 00:07:35.597381 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.07_start_test_client ---
I0409 00:07:35.597504 128371071918080 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240409-0003-aex3b: server_target=xds:///psm-grpc-server:64474 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0409 00:07:35.597739 128371071918080 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0003-aex3b%22;timeRange=2024-04-09T00:07:35.597609Z%2F2024-04-09T00:37:35.597609Z?project=grpc-testing
I0409 00:07:35.599200 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240409-0003-aex3b
labels:
name: psm-interop-client-20240409-0003-aex3b
owner: xds-k8s-interop-test
...

I0409 00:07:35.645614 128371071918080 k8s_base_runner.py:367] Namespace psm-interop-client-20240409-0003-aex3b created
I0409 00:07:35.645797 128371071918080 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0003-aex3b/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:07:36.030748 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0003-aex3b
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:07:36.079225 128371071918080 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0409 00:07:36.086305 128371071918080 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0003-aex3b
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-b889rjbh
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-b889rjbh
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-b889rjbh
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:64474"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0409 00:07:36.123468 128371071918080 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0409 00:07:36.123657 128371071918080 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0409 00:07:37.160911 128371071918080 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8579469b6f-nhhzs']
I0409 00:07:37.161105 128371071918080 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-8579469b6f-nhhzs to start
I0409 00:07:39.210564 128371071918080 k8s_base_runner.py:985] Pod psm-grpc-client-8579469b6f-nhhzs ready, IP: 10.120.9.44
I0409 00:07:39.210771 128371071918080 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-8579469b6f-nhhzs to failover_test/test_app_logs/psm-interop-client-20240409-0003-aex3b_psm-grpc-client-8579469b6f-nhhzs.log
I0409 00:07:39.211188 128370734003776 k8s_log_collector.py:71] [ns/psm-interop-client-20240409-0003-aex3b] Starting log collection thread 128370734003776 for psm-grpc-client-8579469b6f-nhhzs
I0409 00:07:39.211374 128371071918080 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0409 00:07:43.278807 128371071918080 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0409 00:07:43.279138 128371071918080 client_app.py:292] [psm-grpc-client-8579469b6f-nhhzs] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0409 00:07:43.319992 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:07:53.341782 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:08:03.364578 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:08:13.385833 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:08:23.409136 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:08:39.436612 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:09:04.473782 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:09:29.508076 128371071918080 client_app.py:320] [psm-grpc-client-8579469b6f-nhhzs] xDS control plane channel:
I0409 00:09:31.517349 128371071918080 client_app.py:331] [psm-grpc-client-8579469b6f-nhhzs] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0409 00:09:31.517548 128371071918080 client_app.py:302] [psm-grpc-client-8579469b6f-nhhzs] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0409 00:09:31.517725 128371071918080 client_app.py:252] [psm-grpc-client-8579469b6f-nhhzs] Waiting to report a READY channel to xds:///psm-grpc-server:64474
I0409 00:09:31.521306 128371071918080 client_app.py:379] [psm-grpc-client-8579469b6f-nhhzs] Server channel:
I0409 00:09:31.529441 128371071918080 client_app.py:392] [psm-grpc-client-8579469b6f-nhhzs] Found subchannel in state READY:
I0409 00:09:31.529597 128371071918080 client_app.py:263] [psm-grpc-client-8579469b6f-nhhzs] Channel to xds:///psm-grpc-server:64474 transitioned to state READY:
I0409 00:09:31.529684 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.07_start_test_client ---
I0409 00:09:31.529743 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0409 00:09:31.529879 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0409 00:09:31.548659 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0409 00:09:31.548833 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0409 00:09:31.549081 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-4vqbl', 'psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:09:31.549316 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:35.566154 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 33
psm-grpc-server-6c9c6b757b-xzzk6: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 33
psm-grpc-server-6c9c6b757b-xzzk6: 34

I0409 00:09:35.566654 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0409 00:09:35.566770 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0409 00:09:35.566867 128371071918080 server_app.py:108] [psm-grpc-server-6c9c6b757b-4vqbl] >> Setting health status to NOT_SERVING
I0409 00:09:35.567168 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-4vqbl:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:09:35.579109 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-4vqbl:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:09:35.581495 128371071918080 server_app.py:112] [psm-grpc-server-6c9c6b757b-4vqbl] << Health status status: NOT_SERVING

I0409 00:09:35.581757 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:09:35.581869 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:39.595095 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 34
psm-grpc-server-6c9c6b757b-xzzk6: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 34
psm-grpc-server-6c9c6b757b-xzzk6: 33

I0409 00:09:39.595667 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-4vqbl' not found in ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6'] : Unexpected server psm-grpc-server-6c9c6b757b-4vqbl received RPCs
I0409 00:09:40.597177 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:09:40.597699 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:44.614534 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 33
psm-grpc-server-6c9c6b757b-xzzk6: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 33
psm-grpc-server-6c9c6b757b-xzzk6: 34

I0409 00:09:44.614972 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-4vqbl' not found in ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6'] : Unexpected server psm-grpc-server-6c9c6b757b-4vqbl received RPCs
I0409 00:09:45.616244 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:09:45.616539 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:49.618113 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 1
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 49
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 1
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 49

I0409 00:09:49.618586 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-4vqbl' not found in ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6'] : Unexpected server psm-grpc-server-6c9c6b757b-4vqbl received RPCs
I0409 00:09:50.619904 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:09:50.620247 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:54.637703 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50

I0409 00:09:54.638146 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0409 00:09:54.638246 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.11_gentle_failover ---
I0409 00:09:54.638317 128371071918080 server_app.py:108] [psm-grpc-server-6c9c6b757b-gzjqd] >> Setting health status to NOT_SERVING
I0409 00:09:54.638572 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-gzjqd:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:09:54.648062 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-gzjqd:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:09:54.650494 128371071918080 server_app.py:112] [psm-grpc-server-6c9c6b757b-gzjqd] << Health status status: NOT_SERVING

I0409 00:09:54.650984 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-xzzk6', 'psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:09:54.651185 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:09:58.672546 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50

I0409 00:09:58.673197 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-79dbc67778-hmssx' not found in {'psm-grpc-server-6c9c6b757b-gzjqd': 50, 'psm-grpc-server-6c9c6b757b-xzzk6': 50} : Server psm-grpc-server-alt-79dbc67778-hmssx did not receive RPCs
I0409 00:09:59.674473 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-xzzk6', 'psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:09:59.674808 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:03.700970 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-gzjqd: 50
psm-grpc-server-6c9c6b757b-xzzk6: 50

I0409 00:10:03.701315 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-79dbc67778-hmssx' not found in {'psm-grpc-server-6c9c6b757b-gzjqd': 50, 'psm-grpc-server-6c9c6b757b-xzzk6': 50} : Server psm-grpc-server-alt-79dbc67778-hmssx did not receive RPCs
I0409 00:10:04.702735 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-xzzk6', 'psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:10:04.703027 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:08.742454 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-gzjqd: 13
psm-grpc-server-alt-79dbc67778-hmssx: 20
psm-grpc-server-6c9c6b757b-xzzk6: 67
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-gzjqd: 13
psm-grpc-server-alt-79dbc67778-hmssx: 20
psm-grpc-server-6c9c6b757b-xzzk6: 67

I0409 00:10:08.742806 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-gzjqd' not found in ['psm-grpc-server-6c9c6b757b-xzzk6', 'psm-grpc-server-alt-79dbc67778-hmssx'] : Unexpected server psm-grpc-server-6c9c6b757b-gzjqd received RPCs
I0409 00:10:09.744241 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-xzzk6', 'psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:10:09.744552 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:13.763586 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-79dbc67778-hmssx: 28
psm-grpc-server-6c9c6b757b-xzzk6: 72
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-79dbc67778-hmssx: 28
psm-grpc-server-6c9c6b757b-xzzk6: 72

I0409 00:10:13.763870 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.11_gentle_failover ---
I0409 00:10:13.763992 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0409 00:10:13.764040 128371071918080 server_app.py:108] [psm-grpc-server-6c9c6b757b-xzzk6] >> Setting health status to NOT_SERVING
I0409 00:10:13.764275 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-xzzk6:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:10:13.775637 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-xzzk6:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:10:13.778181 128371071918080 server_app.py:112] [psm-grpc-server-6c9c6b757b-xzzk6] << Health status status: NOT_SERVING

I0409 00:10:13.778531 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:10:13.778677 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:17.797655 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-79dbc67778-hmssx: 37
psm-grpc-server-6c9c6b757b-xzzk6: 63
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-79dbc67778-hmssx: 37
psm-grpc-server-6c9c6b757b-xzzk6: 63

I0409 00:10:17.798307 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-xzzk6' not found in ['psm-grpc-server-alt-79dbc67778-hmssx'] : Unexpected server psm-grpc-server-6c9c6b757b-xzzk6 received RPCs
I0409 00:10:18.799592 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:10:18.799953 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:22.860589 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-79dbc67778-hmssx: 52
psm-grpc-server-6c9c6b757b-xzzk6: 48
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-79dbc67778-hmssx: 52
psm-grpc-server-6c9c6b757b-xzzk6: 48

I0409 00:10:22.861043 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-xzzk6' not found in ['psm-grpc-server-alt-79dbc67778-hmssx'] : Unexpected server psm-grpc-server-6c9c6b757b-xzzk6 received RPCs
I0409 00:10:23.862392 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-alt-79dbc67778-hmssx']
I0409 00:10:23.862740 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:27.903793 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-79dbc67778-hmssx: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-79dbc67778-hmssx: 100

I0409 00:10:27.904263 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0409 00:10:27.904407 128371071918080 xds_k8s_testcase.py:299] --- Starting subTest FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0409 00:10:27.904500 128371071918080 server_app.py:99] [psm-grpc-server-6c9c6b757b-4vqbl] >> Setting health status to SERVING
I0409 00:10:27.904611 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-4vqbl:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.910130 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-4vqbl:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.912728 128371071918080 server_app.py:101] [psm-grpc-server-6c9c6b757b-4vqbl] << Health status status: SERVING

I0409 00:10:27.912936 128371071918080 server_app.py:99] [psm-grpc-server-6c9c6b757b-gzjqd] >> Setting health status to SERVING
I0409 00:10:27.913059 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-gzjqd:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.916772 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-gzjqd:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.918966 128371071918080 server_app.py:101] [psm-grpc-server-6c9c6b757b-gzjqd] << Health status status: SERVING

I0409 00:10:27.919052 128371071918080 server_app.py:99] [psm-grpc-server-6c9c6b757b-xzzk6] >> Setting health status to SERVING
I0409 00:10:27.919143 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-xzzk6:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.923579 128371071918080 grpc.py:75] [psm-grpc-server-6c9c6b757b-xzzk6:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0409 00:10:27.925711 128371071918080 server_app.py:101] [psm-grpc-server-6c9c6b757b-xzzk6] << Health status status: SERVING

I0409 00:10:27.926012 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-4vqbl', 'psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:10:27.926134 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:31.962653 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-79dbc67778-hmssx: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-79dbc67778-hmssx: 100

I0409 00:10:31.963328 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-4vqbl' not found in {'psm-grpc-server-alt-79dbc67778-hmssx': 100} : Server psm-grpc-server-6c9c6b757b-4vqbl did not receive RPCs
I0409 00:10:32.964682 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-4vqbl', 'psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:10:32.964993 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:36.963028 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 15
psm-grpc-server-alt-79dbc67778-hmssx: 70
psm-grpc-server-6c9c6b757b-gzjqd: 15
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 15
psm-grpc-server-alt-79dbc67778-hmssx: 70
psm-grpc-server-6c9c6b757b-gzjqd: 15

I0409 00:10:36.963502 128371071918080 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6c9c6b757b-xzzk6' not found in {'psm-grpc-server-6c9c6b757b-4vqbl': 15, 'psm-grpc-server-alt-79dbc67778-hmssx': 70, 'psm-grpc-server-6c9c6b757b-gzjqd': 15} : Server psm-grpc-server-6c9c6b757b-xzzk6 did not receive RPCs
I0409 00:10:37.964777 128371071918080 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-6c9c6b757b-4vqbl', 'psm-grpc-server-6c9c6b757b-gzjqd', 'psm-grpc-server-6c9c6b757b-xzzk6']
I0409 00:10:37.965084 128371071918080 grpc.py:75] [psm-grpc-client-8579469b6f-nhhzs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:10:41.977054 128371071918080 xds_k8s_testcase.py:669] [psm-grpc-client-8579469b6f-nhhzs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 34
psm-grpc-server-6c9c6b757b-xzzk6: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-6c9c6b757b-4vqbl: 33
psm-grpc-server-6c9c6b757b-gzjqd: 34
psm-grpc-server-6c9c6b757b-xzzk6: 33

I0409 00:10:41.977359 128371071918080 xds_k8s_testcase.py:304] --- Finished subTest FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0409 00:10:41.977647 128371071918080 xds_k8s_testcase.py:755] ----- TestMethod FailoverTest.test_failover teardown -----
I0409 00:10:42.071258 128371071918080 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240409-0003-aex3b"
I0409 00:10:42.786464 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621442074-6159ec0b088f1-038c12f7-96f89b12
I0409 00:11:15.582087 128371071918080 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240409-0003-aex3b"
I0409 00:11:15.733381 128371071918080 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240409-0003-aex3b"
I0409 00:11:16.171561 128371071918080 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240409-0003-aex3b"
I0409 00:11:16.924990 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621476174-6159ec2b8d9f5-fa94ff11-e9ec0834
I0409 00:11:19.231137 128371071918080 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240409-0003-aex3b"
I0409 00:11:19.620675 128371071918080 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240409-0003-aex3b"
I0409 00:11:20.303044 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621479629-6159ec2ed917b-81678cbf-b5cdb996
I0409 00:11:22.682606 128371071918080 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240409-0003-aex3b"
I0409 00:11:23.075098 128371071918080 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240409-0003-aex3b"
I0409 00:11:23.664944 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621483084-6159ec32249bb-45896735-4e250dc2
I0409 00:11:54.229797 128371071918080 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240409-0003-aex3b"
I0409 00:11:54.660758 128371071918080 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240409-0003-aex3b"
I0409 00:11:55.121846 128371071918080 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240409-0003-aex3b"
I0409 00:11:55.714659 128371071918080 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621515192-6159ec50c3a86-e58303e9-606b5746
I0409 00:12:00.195263 128371071918080 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0409 00:12:40.324073 128371071918080 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0409 00:12:40.324413 128371071918080 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0003-aex3b/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:12:40.779327 128371071918080 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0409 00:12:40.830711 128371071918080 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0409 00:12:40.830959 128371071918080 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240409-0003-aex3b
I0409 00:13:10.952270 128371071918080 k8s_base_runner.py:941] Namespace psm-interop-client-20240409-0003-aex3b deleted
I0409 00:13:10.952450 128371071918080 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0409 00:13:21.031805 128371071918080 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0409 00:13:21.032018 128371071918080 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0409 00:13:22.087612 128371071918080 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0409 00:13:22.087826 128371071918080 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0003-aex3b/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:13:22.625033 128371071918080 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0409 00:13:22.678204 128371071918080 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0409 00:13:22.678415 128371071918080 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0003-aex3b
I0409 00:13:52.767183 128371071918080 k8s_base_runner.py:941] Namespace psm-interop-server-20240409-0003-aex3b deleted
I0409 00:13:52.767411 128371071918080 k8s_base_runner.py:873] Deleting deployment psm-grpc-server-alt
I0409 00:14:03.129047 128371071918080 k8s_base_runner.py:888] Deployment psm-grpc-server-alt deleted
I0409 00:14:03.129311 128371071918080 k8s_base_runner.py:891] Deleting service psm-grpc-server-alt
I0409 00:14:03.258074 128371071918080 k8s_base_runner.py:904] Service psm-grpc-server-alt deleted
I0409 00:14:03.258289 128371071918080 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0003-aex3b/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:14:03.726451 128371071918080 k8s_base_runner.py:907] Deleting service account psm-grpc-server-alt
I0409 00:14:03.857186 128371071918080 k8s_base_runner.py:921] Service account psm-grpc-server-alt deleted
I0409 00:14:03.857424 128371071918080 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0003-aex3b
I0409 00:16:04.331873 128371071918080 k8s_base_runner.py:941] Namespace psm-interop-server-20240409-0003-aex3b deleted
I0409 00:16:04.332209 128371071918080 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0409 00:16:04.332400 128371071918080 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-b889rjbh:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0003-aex3b%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-b889rjbh%22;timeRange=2024-04-09T00:07:35.597602Z%2F2024-04-09T00:13:10.952425Z;cursorTimestamp=2024-04-09T00:07:43.278993Z?project=grpc-testing
I0409 00:16:04.332537 128371071918080 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-4ni2idyy:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0003-aex3b%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-4ni2idyy%22;timeRange=2024-04-09T00:05:08.940113Z%2F2024-04-09T00:13:52.767384Z;cursorTimestamp=2024-04-09T00:05:20.984955Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
I0409 00:16:04.332773 128371071918080 base_testcase.py:67] ----- PSM Test Case PASSED: FailoverTest.test_failover -----

----------------------------------------------------------------------
Ran 1 test in 748.884s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/outlier_detection_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0409 00:16:05.365105 126343945183232 xds_k8s_testcase.py:162] ----- Testing OutlierDetectionTest -----
I0409 00:16:05.365214 126343945183232 xds_k8s_testcase.py:163] Logs timezone: UTC
I0409 00:16:05.365413 126343945183232 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0409 00:16:05.377716 126343945183232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0409 00:16:05.390186 126343945183232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0409 00:16:05.393894 126343945183232 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240409-0016-ruoqy
I0409 00:16:05.626927 126343945183232 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/outlier_detection_test
I0409 00:16:06.002604 126343945183232 xds_k8s_testcase.py:731] Found unused xds port: 25573
I0409 00:16:06.002871 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0409 00:16:06.002958 126343945183232 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240409-0016-ruoqy"
I0409 00:16:06.005211 126343945183232 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240409-0016-ruoqy
type: GRPC
...

I0409 00:16:06.671404 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621766076-6159ed400673a-1647adbb-42e30359
I0409 00:16:09.008851 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0409 00:16:09.009003 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0409 00:16:09.009066 126343945183232 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240409-0016-ruoqy"
I0409 00:16:09.014689 126343945183232 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240409-0016-ruoqy
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240409-0016-ruoqy
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...

I0409 00:16:09.825616 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621769021-6159ed42d5652-cd4eb494-f4c3dd4f
I0409 00:16:31.540202 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0409 00:16:31.540414 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0409 00:16:31.540504 126343945183232 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240409-0016-ruoqy": psm-grpc-server:25573 -> psm-interop-backend-service-20240409-0016-ruoqy
I0409 00:16:31.547791 126343945183232 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0016-ruoqy
hostRules:
- hosts:
- psm-grpc-server:25573
pathMatcher: psm-interop-path-matcher-20240409-0016-ruoqy
name: psm-interop-url-map-20240409-0016-ruoqy
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0016-ruoqy
name: psm-interop-path-matcher-20240409-0016-ruoqy
...

I0409 00:16:32.185102 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621791554-6159ed5852a80-e7a4b12f-61bbbfdf
I0409 00:16:34.465047 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0409 00:16:34.465232 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0409 00:16:34.465331 126343945183232 traffic_director.py:578] Creating target psm-interop-target-proxy-20240409-0016-ruoqy proxy "GRPC" to URL map psm-interop-url-map-20240409-0016-ruoqy
I0409 00:16:34.466749 126343945183232 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240409-0016-ruoqy
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240409-0016-ruoqy
validate_for_proxyless: true
...

I0409 00:16:35.005738 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621794468-6159ed5b1a332-7fb99384-881b5b3b
I0409 00:16:37.361323 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0409 00:16:37.361506 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0409 00:16:37.361595 126343945183232 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240409-0016-ruoqy" in network "default-vpc": 0.0.0.0:25573 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0016-ruoqy
I0409 00:16:37.363479 126343945183232 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240409-0016-ruoqy
network: global/networks/default-vpc
portRange: 25573
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0016-ruoqy
...

I0409 00:16:38.158468 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621797365-6159ed5ddd691-1ec83028-1ca43aaf
I0409 00:16:51.262163 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0409 00:16:51.262381 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0409 00:16:51.262487 126343945183232 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240409-0016-ruoqy: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0409 00:16:51.262704 126343945183232 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0016-ruoqy%22;timeRange=2024-04-09T00:16:51.262583Z%2F2024-04-09T00:46:51.262583Z?project=grpc-testing
I0409 00:16:51.266553 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240409-0016-ruoqy
labels:
name: psm-interop-server-20240409-0016-ruoqy
owner: xds-k8s-interop-test
...

I0409 00:16:51.332911 126343945183232 k8s_base_runner.py:367] Namespace psm-interop-server-20240409-0016-ruoqy created
I0409 00:16:51.335103 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0016-ruoqy
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0016-ruoqy-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:16:51.369080 126343945183232 k8s_base_runner.py:367] Service psm-grpc-server created
I0409 00:16:51.369246 126343945183232 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240409-0016-ruoqy
I0409 00:16:51.395716 126343945183232 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240409-0016-ruoqy: neg_name=psm-interop-server-20240409-0016-ruoqy-psm-grpc-server, port=8080, zones=['us-central1-a']
I0409 00:16:51.395866 126343945183232 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0016-ruoqy/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:16:51.935285 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0016-ruoqy
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:16:51.958456 126343945183232 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0409 00:16:51.970047 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0016-ruoqy
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zfbfni9t
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zfbfni9t
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-zfbfni9t
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0016-ruoqy-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:16:52.006162 126343945183232 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0409 00:16:52.006355 126343945183232 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0409 00:16:53.087989 126343945183232 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:16:53.088302 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-59d964747c-68d7f to start
I0409 00:16:56.157602 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-server-59d964747c-68d7f ready, IP: 10.120.14.15
I0409 00:16:56.157813 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-59d964747c-68d7f to outlier_detection_test/test_app_logs/psm-interop-server-20240409-0016-ruoqy_psm-grpc-server-59d964747c-68d7f.log
I0409 00:16:56.158318 126343848650304 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0016-ruoqy] Starting log collection thread 126343848650304 for psm-grpc-server-59d964747c-68d7f
I0409 00:16:56.158496 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-59d964747c-7q6w5 to start
I0409 00:16:56.195501 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-server-59d964747c-7q6w5 ready, IP: 10.120.12.62
I0409 00:16:56.195720 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-59d964747c-7q6w5 to outlier_detection_test/test_app_logs/psm-interop-server-20240409-0016-ruoqy_psm-grpc-server-59d964747c-7q6w5.log
I0409 00:16:56.196164 126343840257600 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0016-ruoqy] Starting log collection thread 126343840257600 for psm-grpc-server-59d964747c-7q6w5
I0409 00:16:56.196327 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-59d964747c-jmcqt to start
I0409 00:16:56.231403 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-server-59d964747c-jmcqt ready, IP: 10.120.11.162
I0409 00:16:56.231583 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-59d964747c-jmcqt to outlier_detection_test/test_app_logs/psm-interop-server-20240409-0016-ruoqy_psm-grpc-server-59d964747c-jmcqt.log
I0409 00:16:56.231994 126343831864896 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0016-ruoqy] Starting log collection thread 126343831864896 for psm-grpc-server-59d964747c-jmcqt
I0409 00:16:56.232175 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-59d964747c-nlvw6 to start
I0409 00:16:56.264574 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-server-59d964747c-nlvw6 ready, IP: 10.120.17.90
I0409 00:16:56.265308 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-59d964747c-nlvw6 to outlier_detection_test/test_app_logs/psm-interop-server-20240409-0016-ruoqy_psm-grpc-server-59d964747c-nlvw6.log
I0409 00:16:56.265804 126343823472192 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0016-ruoqy] Starting log collection thread 126343823472192 for psm-grpc-server-59d964747c-nlvw6
I0409 00:16:56.265993 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-59d964747c-wdbjg to start
I0409 00:16:56.298298 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-server-59d964747c-wdbjg ready, IP: 10.120.10.217
I0409 00:16:56.298471 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-59d964747c-wdbjg to outlier_detection_test/test_app_logs/psm-interop-server-20240409-0016-ruoqy_psm-grpc-server-59d964747c-wdbjg.log
I0409 00:16:56.298939 126343815079488 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0016-ruoqy] Starting log collection thread 126343815079488 for psm-grpc-server-59d964747c-wdbjg
I0409 00:16:56.299113 126343945183232 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0409 00:17:00.365753 126343945183232 k8s_base_runner.py:953] Deployment psm-grpc-server has 5 replicas available
I0409 00:17:00.366023 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0409 00:17:00.366117 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0409 00:17:00.370901 126343945183232 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:17:00.466675 126343945183232 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0016-ruoqy-psm-grpc-server" in zone us-central1-a
I0409 00:17:00.466859 126343945183232 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0016-ruoqy: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0016-ruoqy-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0016-ruoqy-psm-grpc-server', zone='us-central1-a')}
I0409 00:17:00.469995 126343945183232 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0016-ruoqy-psm-grpc-server
maxRatePerEndpoint: 5
...

I0409 00:17:01.757481 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712621820472-6159ed73e6d06-6ed3fb1d-d4ae695e
I0409 00:18:00.250588 126343945183232 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0016-ruoqy to report all backends healthy: ['psm-interop-server-20240409-0016-ruoqy-psm-grpc-server']
I0409 00:18:00.823204 126343945183232 compute.py:494] Backend psm-interop-server-20240409-0016-ruoqy-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:18:04.827973 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0409 00:18:04.828186 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0409 00:18:04.828296 126343945183232 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240409-0016-ruoqy: server_target=xds:///psm-grpc-server:25573 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0409 00:18:04.828494 126343945183232 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0016-ruoqy%22;timeRange=2024-04-09T00:18:04.828392Z%2F2024-04-09T00:48:04.828392Z?project=grpc-testing
I0409 00:18:04.830024 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240409-0016-ruoqy
labels:
name: psm-interop-client-20240409-0016-ruoqy
owner: xds-k8s-interop-test
...

I0409 00:18:04.847404 126343945183232 k8s_base_runner.py:367] Namespace psm-interop-client-20240409-0016-ruoqy created
I0409 00:18:04.847640 126343945183232 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0016-ruoqy/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:18:05.339566 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0016-ruoqy
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:18:05.367565 126343945183232 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0409 00:18:05.374642 126343945183232 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0016-ruoqy
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-92acn530
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-92acn530
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-92acn530
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:25573"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0409 00:18:05.415882 126343945183232 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0409 00:18:05.416073 126343945183232 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0409 00:18:05.475404 126343945183232 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-79d9ddc4f4-6b448']
I0409 00:18:05.475684 126343945183232 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-79d9ddc4f4-6b448 to start
I0409 00:18:09.588456 126343945183232 k8s_base_runner.py:985] Pod psm-grpc-client-79d9ddc4f4-6b448 ready, IP: 10.120.15.65
I0409 00:18:09.588668 126343945183232 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-79d9ddc4f4-6b448 to outlier_detection_test/test_app_logs/psm-interop-client-20240409-0016-ruoqy_psm-grpc-client-79d9ddc4f4-6b448.log
I0409 00:18:09.589050 126343806686784 k8s_log_collector.py:71] [ns/psm-interop-client-20240409-0016-ruoqy] Starting log collection thread 126343806686784 for psm-grpc-client-79d9ddc4f4-6b448
I0409 00:18:09.589221 126343945183232 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0409 00:18:12.689943 126343945183232 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0409 00:18:12.690498 126343945183232 client_app.py:292] [psm-grpc-client-79d9ddc4f4-6b448] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0409 00:18:12.732546 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:18:22.752882 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:18:32.774128 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:18:42.795745 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:18:52.817830 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:19:08.843832 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:19:33.878612 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:19:58.918432 126343945183232 client_app.py:320] [psm-grpc-client-79d9ddc4f4-6b448] xDS control plane channel:
I0409 00:20:00.927922 126343945183232 client_app.py:331] [psm-grpc-client-79d9ddc4f4-6b448] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0409 00:20:00.928156 126343945183232 client_app.py:302] [psm-grpc-client-79d9ddc4f4-6b448] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0409 00:20:00.928468 126343945183232 client_app.py:252] [psm-grpc-client-79d9ddc4f4-6b448] Waiting to report a READY channel to xds:///psm-grpc-server:25573
I0409 00:20:00.933104 126343945183232 client_app.py:379] [psm-grpc-client-79d9ddc4f4-6b448] Server channel:
I0409 00:20:00.941639 126343945183232 client_app.py:392] [psm-grpc-client-79d9ddc4f4-6b448] Found subchannel in state READY:
I0409 00:20:00.941880 126343945183232 client_app.py:263] [psm-grpc-client-79d9ddc4f4-6b448] Channel to xds:///psm-grpc-server:25573 transitioned to state READY:
I0409 00:20:00.942092 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0409 00:20:00.942172 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0409 00:20:00.942364 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0409 00:20:00.957357 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0409 00:20:00.957679 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0409 00:20:00.958045 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:00.958233 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:01.960966 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 20
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 20
psm-grpc-server-59d964747c-68d7f: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 20
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 20
psm-grpc-server-59d964747c-68d7f: 20

I0409 00:20:01.961398 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0409 00:20:01.961500 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0409 00:20:01.961786 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-59d964747c-68d7f error-code-2'}]}), wait_for_ready=True, timeout=5)
I0409 00:20:01.969477 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:01.969609 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:02.970258 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 20
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 20
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 20
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 20

I0409 00:20:02.970882 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 20 not less than or equal to 0 : Expected all RPCs to succeed: 20 of 100 failed
I0409 00:20:03.972266 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:03.972590 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:04.974126 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:04.974563 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0409 00:20:04.974668 126343945183232 xds_k8s_testcase.py:299] --- Starting subTest OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0409 00:20:04.974803 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), wait_for_ready=True, timeout=5)
I0409 00:20:04.979942 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:04.980214 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:05.988856 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:05.989368 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:06.990649 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:06.990988 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:08.004455 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:08.004863 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:09.006131 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:09.006431 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:10.012271 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:10.012691 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:11.013891 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:11.014182 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:12.018199 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:12.018628 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:13.019939 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:13.020257 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:14.026789 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:14.027197 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:15.028440 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:15.028739 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:16.029759 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:16.030216 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:17.031475 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:17.031785 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:18.038857 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:18.039286 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:19.040534 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:19.040853 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:20.046120 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:20.046534 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:21.047785 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:21.048122 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:22.047566 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:22.048026 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:23.049294 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:23.049624 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:24.060458 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:24.060935 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:25.062217 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:25.062521 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:26.067915 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:26.068336 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:27.069602 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:27.069887 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:28.080168 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:28.080617 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:29.081892 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:29.082201 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:30.091153 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:30.091576 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:31.092865 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:31.093176 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:32.097404 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 25
psm-grpc-server-59d964747c-nlvw6: 25
psm-grpc-server-59d964747c-jmcqt: 25
psm-grpc-server-59d964747c-wdbjg: 25

I0409 00:20:32.097794 126343945183232 retryers.py:213] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-59d964747c-68d7f' not found in {'psm-grpc-server-59d964747c-7q6w5': 25, 'psm-grpc-server-59d964747c-nlvw6': 25, 'psm-grpc-server-59d964747c-jmcqt': 25, 'psm-grpc-server-59d964747c-wdbjg': 25} : Server psm-grpc-server-59d964747c-68d7f did not receive RPCs
I0409 00:20:33.099068 126343945183232 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-59d964747c-68d7f', 'psm-grpc-server-59d964747c-7q6w5', 'psm-grpc-server-59d964747c-jmcqt', 'psm-grpc-server-59d964747c-nlvw6', 'psm-grpc-server-59d964747c-wdbjg']
I0409 00:20:33.099381 126343945183232 grpc.py:75] [psm-grpc-client-79d9ddc4f4-6b448:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:20:34.090825 126343945183232 xds_k8s_testcase.py:669] [psm-grpc-client-79d9ddc4f4-6b448] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 21
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 21
psm-grpc-server-59d964747c-68d7f: 18
rpcs_by_method:
UnaryCall:
psm-grpc-server-59d964747c-7q6w5: 20
psm-grpc-server-59d964747c-nlvw6: 21
psm-grpc-server-59d964747c-jmcqt: 20
psm-grpc-server-59d964747c-wdbjg: 21
psm-grpc-server-59d964747c-68d7f: 18

I0409 00:20:34.091116 126343945183232 xds_k8s_testcase.py:304] --- Finished subTest OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0409 00:20:34.091230 126343945183232 xds_k8s_testcase.py:755] ----- TestMethod OutlierDetectionTest.test_outlier_detection teardown -----
I0409 00:20:34.194813 126343945183232 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240409-0016-ruoqy"
I0409 00:20:34.961932 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622034202-6159ee3fbafba-27bb41e1-13ae2de0
I0409 00:21:07.418342 126343945183232 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240409-0016-ruoqy"
I0409 00:21:07.502485 126343945183232 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240409-0016-ruoqy"
I0409 00:21:07.966933 126343945183232 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240409-0016-ruoqy"
I0409 00:21:08.425875 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622067974-6159ee5ff0000-51a15382-ac6c7fec
I0409 00:21:10.704468 126343945183232 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240409-0016-ruoqy"
I0409 00:21:11.223575 126343945183232 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240409-0016-ruoqy"
I0409 00:21:11.905837 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622071228-6159ee630a718-0cc03909-9597bd33
I0409 00:21:14.278117 126343945183232 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240409-0016-ruoqy"
I0409 00:21:14.656385 126343945183232 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240409-0016-ruoqy"
I0409 00:21:15.449325 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622074660-6159ee665066d-aa641dde-f1c40201
I0409 00:21:43.578458 126343945183232 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240409-0016-ruoqy"
I0409 00:21:43.941691 126343945183232 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240409-0016-ruoqy"
I0409 00:21:44.376067 126343945183232 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240409-0016-ruoqy"
I0409 00:21:44.907921 126343945183232 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622104462-6159ee82bc464-0b8e3fbe-ced79238
I0409 00:21:49.336065 126343945183232 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0409 00:22:29.506534 126343945183232 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0409 00:22:29.506797 126343945183232 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0016-ruoqy/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:22:29.941875 126343945183232 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0409 00:22:30.003039 126343945183232 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0409 00:22:30.003257 126343945183232 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240409-0016-ruoqy
I0409 00:23:00.784808 126343945183232 k8s_base_runner.py:941] Namespace psm-interop-client-20240409-0016-ruoqy deleted
I0409 00:23:00.784989 126343945183232 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0409 00:23:10.877627 126343945183232 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0409 00:23:10.877897 126343945183232 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0409 00:23:11.946013 126343945183232 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0409 00:23:11.946217 126343945183232 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0016-ruoqy/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:23:12.457208 126343945183232 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0409 00:23:12.503665 126343945183232 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0409 00:23:12.503893 126343945183232 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0016-ruoqy
I0409 00:24:12.627210 126343945183232 k8s_base_runner.py:941] Namespace psm-interop-server-20240409-0016-ruoqy deleted
I0409 00:24:12.627600 126343945183232 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0409 00:24:12.627824 126343945183232 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-92acn530:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0016-ruoqy%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-92acn530%22;timeRange=2024-04-09T00:18:04.828385Z%2F2024-04-09T00:23:00.784963Z;cursorTimestamp=2024-04-09T00:18:12.690302Z?project=grpc-testing
I0409 00:24:12.627980 126343945183232 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-zfbfni9t:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0016-ruoqy%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-zfbfni9t%22;timeRange=2024-04-09T00:16:51.262576Z%2F2024-04-09T00:24:12.627458Z;cursorTimestamp=2024-04-09T00:17:00.365956Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
I0409 00:24:12.628233 126343945183232 base_testcase.py:67] ----- PSM Test Case PASSED: OutlierDetectionTest.test_outlier_detection -----

----------------------------------------------------------------------
Ran 1 test in 487.266s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/remove_neg_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0409 00:24:13.713644 129182709268480 xds_k8s_testcase.py:162] ----- Testing RemoveNegTest -----
I0409 00:24:13.713762 129182709268480 xds_k8s_testcase.py:163] Logs timezone: UTC
I0409 00:24:13.713932 129182709268480 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0409 00:24:13.728554 129182709268480 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0409 00:24:13.741452 129182709268480 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0409 00:24:13.745183 129182709268480 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240409-0024-128mi
I0409 00:24:14.097912 129182709268480 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/remove_neg_test
I0409 00:24:14.380210 129182709268480 xds_k8s_testcase.py:731] Found unused xds port: 7457
I0409 00:24:14.382033 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.00_create_health_check ---
I0409 00:24:14.382168 129182709268480 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240409-0024-128mi"
I0409 00:24:14.384248 129182709268480 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240409-0024-128mi
type: GRPC
...

I0409 00:24:15.106342 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622254446-6159ef11c56c7-5982e3f6-455f7bc5
I0409 00:24:17.380087 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.00_create_health_check ---
I0409 00:24:17.380296 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0409 00:24:17.380410 129182709268480 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240409-0024-128mi"
I0409 00:24:17.385959 129182709268480 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240409-0024-128mi
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240409-0024-128mi
protocol: GRPC
...

I0409 00:24:18.123744 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622257392-6159ef14949d8-f75bd401-95f14a5e
I0409 00:24:37.713072 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0409 00:24:37.713277 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.02_create_url_map ---
I0409 00:24:37.713371 129182709268480 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240409-0024-128mi": psm-grpc-server:7457 -> psm-interop-backend-service-20240409-0024-128mi
I0409 00:24:37.720757 129182709268480 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0024-128mi
hostRules:
- hosts:
- psm-grpc-server:7457
pathMatcher: psm-interop-path-matcher-20240409-0024-128mi
name: psm-interop-url-map-20240409-0024-128mi
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0024-128mi
name: psm-interop-path-matcher-20240409-0024-128mi
...

I0409 00:24:38.421653 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622277722-6159ef27f82e0-5d691d42-147cb9d3
I0409 00:24:40.719680 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.02_create_url_map ---
I0409 00:24:40.719903 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0409 00:24:40.720006 129182709268480 traffic_director.py:578] Creating target psm-interop-target-proxy-20240409-0024-128mi proxy "GRPC" to URL map psm-interop-url-map-20240409-0024-128mi
I0409 00:24:40.721455 129182709268480 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240409-0024-128mi
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240409-0024-128mi
validate_for_proxyless: true
...

I0409 00:24:41.345240 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622280723-6159ef2ad4cc2-81498bd0-7ba7ee3c
I0409 00:24:43.653396 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0409 00:24:43.653587 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0409 00:24:43.653688 129182709268480 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240409-0024-128mi" in network "default-vpc": 0.0.0.0:7457 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0024-128mi
I0409 00:24:43.655640 129182709268480 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240409-0024-128mi
network: global/networks/default-vpc
portRange: 7457
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0024-128mi
...

I0409 00:24:44.462182 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622283659-6159ef2da195b-73d46e3f-052f0bc0
I0409 00:24:57.476964 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0409 00:24:57.477168 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0409 00:24:57.477287 129182709268480 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240409-0024-128mi: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0409 00:24:57.477509 129182709268480 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0024-128mi%22;timeRange=2024-04-09T00:24:57.477389Z%2F2024-04-09T00:54:57.477389Z?project=grpc-testing
I0409 00:24:57.481221 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240409-0024-128mi
labels:
name: psm-interop-server-20240409-0024-128mi
owner: xds-k8s-interop-test
...

I0409 00:24:57.517830 129182709268480 k8s_base_runner.py:367] Namespace psm-interop-server-20240409-0024-128mi created
I0409 00:24:57.519991 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0024-128mi
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0024-128mi-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:24:57.556258 129182709268480 k8s_base_runner.py:367] Service psm-grpc-server created
I0409 00:24:57.556470 129182709268480 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240409-0024-128mi
I0409 00:24:57.585302 129182709268480 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240409-0024-128mi: neg_name=psm-interop-server-20240409-0024-128mi-psm-grpc-server, port=8080, zones=['us-central1-a']
I0409 00:24:57.585487 129182709268480 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0024-128mi/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:24:58.133928 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0024-128mi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:24:58.164999 129182709268480 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0409 00:24:58.176523 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0024-128mi
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-obm2ljj0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-obm2ljj0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-obm2ljj0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0024-128mi-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:24:58.219157 129182709268480 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0409 00:24:58.219354 129182709268480 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0409 00:24:58.261262 129182709268480 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5c8bfc8569-kxr8j']
I0409 00:24:58.261468 129182709268480 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-5c8bfc8569-kxr8j to start
I0409 00:25:02.344154 129182709268480 k8s_base_runner.py:985] Pod psm-grpc-server-5c8bfc8569-kxr8j ready, IP: 10.120.15.66
I0409 00:25:02.344373 129182709268480 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-5c8bfc8569-kxr8j to remove_neg_test/test_app_logs/psm-interop-server-20240409-0024-128mi_psm-grpc-server-5c8bfc8569-kxr8j.log
I0409 00:25:02.344720 129182612317760 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0024-128mi] Starting log collection thread 129182612317760 for psm-grpc-server-5c8bfc8569-kxr8j
I0409 00:25:02.344798 129182709268480 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0409 00:25:07.418262 129182709268480 k8s_base_runner.py:953] Deployment psm-grpc-server has 1 replicas available
I0409 00:25:07.418476 129182709268480 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240409-0024-128mi: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0409 00:25:07.418641 129182709268480 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0024-128mi%22;timeRange=2024-04-09T00:25:07.418562Z%2F2024-04-09T00:55:07.418562Z?project=grpc-testing
I0409 00:25:07.418706 129182709268480 k8s_base_runner.py:377] Reusing namespace: psm-interop-server-20240409-0024-128mi
I0409 00:25:07.440330 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0024-128mi
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:25:07.455219 129182709268480 k8s_base_runner.py:367] Service psm-grpc-server-alt created
I0409 00:25:07.455320 129182709268480 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240409-0024-128mi
I0409 00:25:07.485069 129182709268480 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240409-0024-128mi: neg_name=psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0409 00:25:07.485249 129182709268480 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0024-128mi/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:25:08.037419 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0024-128mi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:25:08.076846 129182709268480 k8s_base_runner.py:367] ServiceAccount psm-grpc-server-alt created
I0409 00:25:08.079125 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240409-0024-128mi
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ml1fnk8w
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ml1fnk8w
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ml1fnk8w
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:25:08.110399 129182709268480 k8s_base_runner.py:367] Deployment psm-grpc-server-alt created
I0409 00:25:08.110596 129182709268480 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0409 00:25:08.141166 129182709268480 k8s_base_runner.py:972] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7bdc9d7dc-hpr6h']
I0409 00:25:08.141326 129182709268480 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-alt-7bdc9d7dc-hpr6h to start
I0409 00:25:11.205112 129182709268480 k8s_base_runner.py:985] Pod psm-grpc-server-alt-7bdc9d7dc-hpr6h ready, IP: 10.120.11.164
I0409 00:25:11.205306 129182709268480 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-alt-7bdc9d7dc-hpr6h to remove_neg_test/test_app_logs/psm-interop-server-20240409-0024-128mi_psm-grpc-server-alt-7bdc9d7dc-hpr6h.log
I0409 00:25:11.205775 129182603925056 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0024-128mi] Starting log collection thread 129182603925056 for psm-grpc-server-alt-7bdc9d7dc-hpr6h
I0409 00:25:11.205954 129182709268480 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0409 00:25:18.307605 129182709268480 k8s_base_runner.py:953] Deployment psm-grpc-server-alt has 1 replicas available
I0409 00:25:18.307835 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0409 00:25:18.307925 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0409 00:25:18.336098 129182709268480 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:25:18.413004 129182709268480 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0024-128mi-psm-grpc-server" in zone us-central1-a
I0409 00:25:18.413212 129182709268480 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0024-128mi: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0024-128mi-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server', zone='us-central1-a')}
I0409 00:25:18.416486 129182709268480 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server
maxRatePerEndpoint: 5
...

I0409 00:25:19.490533 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622318418-6159ef4ec79aa-1c68e922-41153cc4
I0409 00:26:13.584784 129182709268480 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0024-128mi to report all backends healthy: ['psm-interop-server-20240409-0024-128mi-psm-grpc-server']
I0409 00:26:17.980089 129182709268480 compute.py:494] Backend psm-interop-server-20240409-0024-128mi-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:26:22.009032 129182709268480 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:26:22.104098 129182709268480 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt" in zone us-central1-a
I0409 00:26:22.104290 129182709268480 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0024-128mi: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0024-128mi-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server', zone='us-central1-a')}
I0409 00:26:22.107820 129182709268480 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server
maxRatePerEndpoint: 5
...

I0409 00:26:23.025153 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622382109-6159ef8b853cb-6efa0986-15feef49
I0409 00:27:12.698449 129182709268480 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0024-128mi to report all backends healthy: ['psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt', 'psm-interop-server-20240409-0024-128mi-psm-grpc-server']
I0409 00:27:13.269585 129182709268480 compute.py:494] Backend psm-interop-server-20240409-0024-128mi-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:27:21.797174 129182709268480 compute.py:494] Backend psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt in zone us-central1-a reported healthy
I0409 00:27:25.801963 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0409 00:27:25.802198 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.07_start_test_client ---
I0409 00:27:25.802332 129182709268480 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240409-0024-128mi: server_target=xds:///psm-grpc-server:7457 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0409 00:27:25.802591 129182709268480 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0024-128mi%22;timeRange=2024-04-09T00:27:25.802444Z%2F2024-04-09T00:57:25.802444Z?project=grpc-testing
I0409 00:27:25.804210 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240409-0024-128mi
labels:
name: psm-interop-client-20240409-0024-128mi
owner: xds-k8s-interop-test
...

I0409 00:27:25.850718 129182709268480 k8s_base_runner.py:367] Namespace psm-interop-client-20240409-0024-128mi created
I0409 00:27:25.850893 129182709268480 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0024-128mi/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:27:26.203054 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0024-128mi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:27:26.234154 129182709268480 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0409 00:27:26.241576 129182709268480 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0024-128mi
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s0am5bmb
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s0am5bmb
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s0am5bmb
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:7457"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0409 00:27:26.286484 129182709268480 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0409 00:27:26.286695 129182709268480 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0409 00:27:26.320472 129182709268480 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8b4b9695c-q2hr8']
I0409 00:27:26.320797 129182709268480 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-8b4b9695c-q2hr8 to start
I0409 00:27:29.387868 129182709268480 k8s_base_runner.py:985] Pod psm-grpc-client-8b4b9695c-q2hr8 ready, IP: 10.120.17.93
I0409 00:27:29.388075 129182709268480 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-8b4b9695c-q2hr8 to remove_neg_test/test_app_logs/psm-interop-client-20240409-0024-128mi_psm-grpc-client-8b4b9695c-q2hr8.log
I0409 00:27:29.388574 129182595532352 k8s_log_collector.py:71] [ns/psm-interop-client-20240409-0024-128mi] Starting log collection thread 129182595532352 for psm-grpc-client-8b4b9695c-q2hr8
I0409 00:27:29.388754 129182709268480 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0409 00:27:33.455307 129182709268480 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0409 00:27:33.455682 129182709268480 client_app.py:292] [psm-grpc-client-8b4b9695c-q2hr8] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0409 00:27:33.500305 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:27:43.524921 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:27:53.547079 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:28:03.568366 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:28:13.593095 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:28:29.619376 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:28:54.656454 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:29:19.692463 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:29:44.728053 129182709268480 client_app.py:320] [psm-grpc-client-8b4b9695c-q2hr8] xDS control plane channel:
I0409 00:29:46.737076 129182709268480 client_app.py:331] [psm-grpc-client-8b4b9695c-q2hr8] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0409 00:29:46.737279 129182709268480 client_app.py:302] [psm-grpc-client-8b4b9695c-q2hr8] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0409 00:29:46.737476 129182709268480 client_app.py:252] [psm-grpc-client-8b4b9695c-q2hr8] Waiting to report a READY channel to xds:///psm-grpc-server:7457
I0409 00:29:46.740668 129182709268480 client_app.py:379] [psm-grpc-client-8b4b9695c-q2hr8] Server channel:
I0409 00:29:46.749645 129182709268480 client_app.py:392] [psm-grpc-client-8b4b9695c-q2hr8] Found subchannel in state READY:
I0409 00:29:46.749994 129182709268480 client_app.py:263] [psm-grpc-client-8b4b9695c-q2hr8] Channel to xds:///psm-grpc-server:7457 transitioned to state READY:
I0409 00:29:46.750116 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.07_start_test_client ---
I0409 00:29:46.750173 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0409 00:29:46.750342 129182709268480 grpc.py:75] [psm-grpc-client-8b4b9695c-q2hr8:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=30, wait_for_ready=True)
I0409 00:29:46.764593 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0409 00:29:46.764732 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0409 00:29:46.764909 129182709268480 grpc.py:75] [psm-grpc-client-8b4b9695c-q2hr8:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:29:50.823963 129182709268480 xds_k8s_testcase.py:669] [psm-grpc-client-8b4b9695c-q2hr8] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5c8bfc8569-kxr8j: 59
psm-grpc-server-alt-7bdc9d7dc-hpr6h: 41
rpcs_by_method:
UnaryCall:
psm-grpc-server-5c8bfc8569-kxr8j: 59
psm-grpc-server-alt-7bdc9d7dc-hpr6h: 41

I0409 00:29:50.824206 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0409 00:29:50.824284 129182709268480 xds_k8s_testcase.py:299] --- Starting subTest RemoveNegTest.test_remove_neg.10_remove_neg ---
I0409 00:29:50.824554 129182709268480 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-5c8bfc8569-kxr8j', 'psm-grpc-server-alt-7bdc9d7dc-hpr6h']
I0409 00:29:50.824770 129182709268480 grpc.py:75] [psm-grpc-client-8b4b9695c-q2hr8:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:29:54.876929 129182709268480 xds_k8s_testcase.py:669] [psm-grpc-client-8b4b9695c-q2hr8] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5c8bfc8569-kxr8j: 49
psm-grpc-server-alt-7bdc9d7dc-hpr6h: 51
rpcs_by_method:
UnaryCall:
psm-grpc-server-5c8bfc8569-kxr8j: 49
psm-grpc-server-alt-7bdc9d7dc-hpr6h: 51

I0409 00:29:54.918089 129182709268480 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:29:54.982837 129182709268480 traffic_director.py:268] Loaded NEG "psm-interop-server-20240409-0024-128mi-psm-grpc-server-alt" in zone us-central1-a
I0409 00:29:54.983033 129182709268480 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0024-128mi: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0024-128mi-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server', zone='us-central1-a')}
I0409 00:29:54.986198 129182709268480 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0024-128mi-psm-grpc-server
maxRatePerEndpoint: 5
...

I0409 00:29:55.943634 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622594988-6159f05689886-601ca5d8-378d9bfd
I0409 00:30:47.762347 129182709268480 xds_k8s_testcase.py:506] Verifying RPCs go to servers ['psm-grpc-server-5c8bfc8569-kxr8j']
I0409 00:30:47.762636 129182709268480 grpc.py:75] [psm-grpc-client-8b4b9695c-q2hr8:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0409 00:30:51.796443 129182709268480 xds_k8s_testcase.py:669] [psm-grpc-client-8b4b9695c-q2hr8] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5c8bfc8569-kxr8j: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5c8bfc8569-kxr8j: 100

I0409 00:30:51.796981 129182709268480 xds_k8s_testcase.py:304] --- Finished subTest RemoveNegTest.test_remove_neg.10_remove_neg ---
I0409 00:30:51.797123 129182709268480 xds_k8s_testcase.py:755] ----- TestMethod RemoveNegTest.test_remove_neg teardown -----
I0409 00:30:51.851282 129182709268480 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240409-0024-128mi"
I0409 00:30:52.475115 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622651858-6159f08cc602e-dee544f9-9dc8688d
I0409 00:31:05.637621 129182709268480 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240409-0024-128mi"
I0409 00:31:05.777967 129182709268480 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240409-0024-128mi"
I0409 00:31:06.155765 129182709268480 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240409-0024-128mi"
I0409 00:31:06.711768 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622666158-6159f09a6926e-1254c3be-b268912d
I0409 00:31:09.030683 129182709268480 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240409-0024-128mi"
I0409 00:31:09.351200 129182709268480 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240409-0024-128mi"
I0409 00:31:10.154567 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622669355-6159f09d75a0d-0f32ada9-8762c109
I0409 00:31:12.456029 129182709268480 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240409-0024-128mi"
I0409 00:31:12.876081 129182709268480 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240409-0024-128mi"
I0409 00:31:13.488567 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622672881-6159f0a0d2893-83754d87-3f1c1739
I0409 00:31:46.005885 129182709268480 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240409-0024-128mi"
I0409 00:31:46.347849 129182709268480 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240409-0024-128mi"
I0409 00:31:46.794700 129182709268480 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240409-0024-128mi"
I0409 00:31:47.277773 129182709268480 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622706873-6159f0c13d5d0-7a31d4f5-da1de449
I0409 00:31:51.779245 129182709268480 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0409 00:32:31.948177 129182709268480 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0409 00:32:31.948446 129182709268480 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0024-128mi/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:32:32.421526 129182709268480 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0409 00:32:32.467761 129182709268480 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0409 00:32:32.467905 129182709268480 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240409-0024-128mi
I0409 00:33:02.542793 129182709268480 k8s_base_runner.py:941] Namespace psm-interop-client-20240409-0024-128mi deleted
I0409 00:33:02.543033 129182709268480 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0409 00:33:12.626716 129182709268480 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0409 00:33:12.626957 129182709268480 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0409 00:33:13.693564 129182709268480 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0409 00:33:13.693761 129182709268480 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0024-128mi/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:33:14.173176 129182709268480 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0409 00:33:14.217895 129182709268480 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0409 00:33:14.218085 129182709268480 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0024-128mi
I0409 00:34:44.392069 129182709268480 k8s_base_runner.py:941] Namespace psm-interop-server-20240409-0024-128mi deleted
I0409 00:34:44.392217 129182709268480 k8s_base_runner.py:873] Deleting deployment psm-grpc-server-alt
I0409 00:34:44.415863 129182709268480 k8s_base_runner.py:891] Deleting service psm-grpc-server-alt
I0409 00:34:44.440812 129182709268480 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0024-128mi/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:34:44.885863 129182709268480 k8s_base_runner.py:907] Deleting service account psm-grpc-server-alt
I0409 00:34:44.909649 129182709268480 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0024-128mi
I0409 00:34:44.931809 129182709268480 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0409 00:34:44.932117 129182709268480 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-s0am5bmb:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0024-128mi%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-s0am5bmb%22;timeRange=2024-04-09T00:27:25.802437Z%2F2024-04-09T00:33:02.543008Z;cursorTimestamp=2024-04-09T00:27:33.455525Z?project=grpc-testing
I0409 00:34:44.932269 129182709268480 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-obm2ljj0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0024-128mi%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-obm2ljj0%22;timeRange=2024-04-09T00:24:57.477380Z%2F2024-04-09T00:34:44.392198Z;cursorTimestamp=2024-04-09T00:25:07.418437Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
I0409 00:34:44.932520 129182709268480 base_testcase.py:67] ----- PSM Test Case PASSED: RemoveNegTest.test_remove_neg -----

----------------------------------------------------------------------
Ran 1 test in 631.221s

OK
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/round_robin_test'
Running tests under Python 3.10.12: /tmp/tmp.S3tOHHXyEN/psm-interop/venv/bin/python3
I0409 00:34:45.924609 130972906758144 xds_k8s_testcase.py:162] ----- Testing RoundRobinTest -----
I0409 00:34:45.924717 130972906758144 xds_k8s_testcase.py:163] Logs timezone: UTC
I0409 00:34:45.924880 130972906758144 xds_k8s_testcase.py:111] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-master')
I0409 00:34:45.936717 130972906758144 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0409 00:34:45.948228 130972906758144 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0409 00:34:45.951676 130972906758144 xds_k8s_testcase.py:736] Test run resource prefix: psm-interop, suffix: 20240409-0034-1pism
I0409 00:34:46.151884 130972906758144 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/master/xds_k8s_lb/round_robin_test
I0409 00:34:46.428486 130972906758144 xds_k8s_testcase.py:731] Found unused xds port: 34030
I0409 00:34:46.428834 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.00_create_health_check ---
I0409 00:34:46.428919 130972906758144 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240409-0034-1pism"
I0409 00:34:46.431217 130972906758144 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240409-0034-1pism
type: GRPC
...

I0409 00:34:47.198293 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622886489-6159f16c88e7e-d91231d2-bb8ae30c
I0409 00:34:49.626958 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.00_create_health_check ---
I0409 00:34:49.627097 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.01_create_backend_services ---
I0409 00:34:49.627156 130972906758144 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240409-0034-1pism"
I0409 00:34:49.632438 130972906758144 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240409-0034-1pism
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240409-0034-1pism
protocol: GRPC
...

I0409 00:34:52.312603 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622889635-6159f16f88d20-c0ed7ab3-1a16609b
I0409 00:35:16.129977 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.01_create_backend_services ---
I0409 00:35:16.130148 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.02_create_url_map ---
I0409 00:35:16.130234 130972906758144 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240409-0034-1pism": psm-grpc-server:34030 -> psm-interop-backend-service-20240409-0034-1pism
I0409 00:35:16.137480 130972906758144 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0034-1pism
hostRules:
- hosts:
- psm-grpc-server:34030
pathMatcher: psm-interop-path-matcher-20240409-0034-1pism
name: psm-interop-url-map-20240409-0034-1pism
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240409-0034-1pism
name: psm-interop-path-matcher-20240409-0034-1pism
...

I0409 00:35:16.814635 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622916139-6159f188cf9de-38320506-7834ccba
I0409 00:35:19.102341 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.02_create_url_map ---
I0409 00:35:19.102524 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0409 00:35:19.102615 130972906758144 traffic_director.py:578] Creating target psm-interop-target-proxy-20240409-0034-1pism proxy "GRPC" to URL map psm-interop-url-map-20240409-0034-1pism
I0409 00:35:19.104055 130972906758144 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240409-0034-1pism
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240409-0034-1pism
validate_for_proxyless: true
...

I0409 00:35:19.613810 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622919105-6159f18ba3cac-b8052b06-a95b1f1b
I0409 00:35:21.947996 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0409 00:35:21.948212 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0409 00:35:21.948322 130972906758144 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240409-0034-1pism" in network "default-vpc": 0.0.0.0:34030 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0034-1pism
I0409 00:35:21.950120 130972906758144 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240409-0034-1pism
network: global/networks/default-vpc
portRange: 34030
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240409-0034-1pism
...

I0409 00:35:22.791034 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622921952-6159f18e5ab2f-ab48e0aa-56edfaa0
I0409 00:35:35.865773 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0409 00:35:35.865963 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.05_start_test_servers ---
I0409 00:35:35.866059 130972906758144 k8s_xds_server_runner.py:186] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240409-0034-1pism: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0409 00:35:35.866281 130972906758144 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0034-1pism%22;timeRange=2024-04-09T00:35:35.866168Z%2F2024-04-09T01:05:35.866168Z?project=grpc-testing
I0409 00:35:35.869927 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240409-0034-1pism
labels:
name: psm-interop-server-20240409-0034-1pism
owner: xds-k8s-interop-test
...

I0409 00:35:35.901215 130972906758144 k8s_base_runner.py:367] Namespace psm-interop-server-20240409-0034-1pism created
I0409 00:35:35.903013 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0034-1pism
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240409-0034-1pism-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0409 00:35:35.944443 130972906758144 k8s_base_runner.py:367] Service psm-grpc-server created
I0409 00:35:35.944538 130972906758144 k8s_base_runner.py:1054] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240409-0034-1pism
I0409 00:35:35.974443 130972906758144 k8s_base_runner.py:1068] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240409-0034-1pism: neg_name=psm-interop-server-20240409-0034-1pism-psm-grpc-server, port=8080, zones=['us-central1-a']
I0409 00:35:35.974534 130972906758144 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0034-1pism/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:35:36.438568 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0034-1pism
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:35:36.465959 130972906758144 k8s_base_runner.py:367] ServiceAccount psm-grpc-server created
I0409 00:35:36.476784 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240409-0034-1pism
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1586e550
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1586e550
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1586e550
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: us-docker.pkg.dev/grpc-testing/psm-interop/java-server:canonical
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000

args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true

resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240409-0034-1pism-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf

volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory

...

I0409 00:35:36.517861 130972906758144 k8s_base_runner.py:367] Deployment psm-grpc-server created
I0409 00:35:36.517999 130972906758144 k8s_base_runner.py:962] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0409 00:35:37.576580 130972906758144 k8s_base_runner.py:972] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-cf69bd678-5gjv5', 'psm-grpc-server-cf69bd678-sjm2h']
I0409 00:35:37.576829 130972906758144 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-cf69bd678-5gjv5 to start
I0409 00:35:40.646319 130972906758144 k8s_base_runner.py:985] Pod psm-grpc-server-cf69bd678-5gjv5 ready, IP: 10.120.9.50
I0409 00:35:40.646520 130972906758144 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-cf69bd678-5gjv5 to round_robin_test/test_app_logs/psm-interop-server-20240409-0034-1pism_psm-grpc-server-cf69bd678-5gjv5.log
I0409 00:35:40.646988 130972810470976 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0034-1pism] Starting log collection thread 130972810470976 for psm-grpc-server-cf69bd678-5gjv5
I0409 00:35:40.647153 130972906758144 k8s_base_runner.py:982] Waiting for pod psm-grpc-server-cf69bd678-sjm2h to start
I0409 00:35:40.682576 130972906758144 k8s_base_runner.py:985] Pod psm-grpc-server-cf69bd678-sjm2h ready, IP: 10.120.14.19
I0409 00:35:40.682731 130972906758144 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-server-cf69bd678-sjm2h to round_robin_test/test_app_logs/psm-interop-server-20240409-0034-1pism_psm-grpc-server-cf69bd678-sjm2h.log
I0409 00:35:40.683051 130972802078272 k8s_log_collector.py:71] [ns/psm-interop-server-20240409-0034-1pism] Starting log collection thread 130972802078272 for psm-grpc-server-cf69bd678-sjm2h
I0409 00:35:40.683188 130972906758144 k8s_base_runner.py:944] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0409 00:35:45.792611 130972906758144 k8s_base_runner.py:953] Deployment psm-grpc-server has 2 replicas available
I0409 00:35:45.792873 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.05_start_test_servers ---
I0409 00:35:45.792962 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0409 00:35:45.801731 130972906758144 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0409 00:35:45.890519 130972906758144 traffic_director.py:259] Loaded NEG "psm-interop-server-20240409-0034-1pism-psm-grpc-server" in zone us-central1-a
I0409 00:35:45.890703 130972906758144 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240409-0034-1pism: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240409-0034-1pism-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0034-1pism-psm-grpc-server', zone='us-central1-a')}
I0409 00:35:45.893654 130972906758144 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240409-0034-1pism-psm-grpc-server
maxRatePerEndpoint: 5
...

I0409 00:35:46.923197 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712622945895-6159f1a5305f0-c45f117d-913a0ea2
I0409 00:36:38.698672 130972906758144 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240409-0034-1pism to report all backends healthy: ['psm-interop-server-20240409-0034-1pism-psm-grpc-server']
I0409 00:36:38.833594 130972906758144 compute.py:494] Backend psm-interop-server-20240409-0034-1pism-psm-grpc-server in zone us-central1-a reported healthy
I0409 00:36:42.838346 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0409 00:36:42.838551 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.07_start_test_client ---
I0409 00:36:42.838661 130972906758144 k8s_xds_client_runner.py:119] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240409-0034-1pism: server_target=xds:///psm-grpc-server:34030 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0409 00:36:42.838884 130972906758144 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0034-1pism%22;timeRange=2024-04-09T00:36:42.838767Z%2F2024-04-09T01:06:42.838767Z?project=grpc-testing
I0409 00:36:42.840433 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240409-0034-1pism
labels:
name: psm-interop-client-20240409-0034-1pism
owner: xds-k8s-interop-test
...

I0409 00:36:42.939150 130972906758144 k8s_base_runner.py:367] Namespace psm-interop-client-20240409-0034-1pism created
I0409 00:36:42.939349 130972906758144 k8s_base_runner.py:452] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0034-1pism/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:36:43.324912 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0034-1pism
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0409 00:36:43.418754 130972906758144 k8s_base_runner.py:367] ServiceAccount psm-grpc-client created
I0409 00:36:43.425319 130972906758144 k8s_base_runner.py:347] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240409-0034-1pism
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-dmrvslxa
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-dmrvslxa
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-dmrvslxa
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: us-docker.pkg.dev/grpc-testing/psm-interop/node-client:ade39e8baa9c327626f83b862195dbb79ea722a0
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:34030"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:2bf1b5ed00f852ffea8d24759c6fa673acc9ef10
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0409 00:36:43.458793 130972906758144 k8s_base_runner.py:367] Deployment psm-grpc-client created
I0409 00:36:43.458926 130972906758144 k8s_base_runner.py:962] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0409 00:36:44.498028 130972906758144 k8s_base_runner.py:972] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6899fdb4c6-vzmlg']
I0409 00:36:44.498237 130972906758144 k8s_base_runner.py:982] Waiting for pod psm-grpc-client-6899fdb4c6-vzmlg to start
I0409 00:36:47.566254 130972906758144 k8s_base_runner.py:985] Pod psm-grpc-client-6899fdb4c6-vzmlg ready, IP: 10.120.17.95
I0409 00:36:47.566451 130972906758144 k8s_base_runner.py:1030] Enabling log collection from pod psm-grpc-client-6899fdb4c6-vzmlg to round_robin_test/test_app_logs/psm-interop-client-20240409-0034-1pism_psm-grpc-client-6899fdb4c6-vzmlg.log
I0409 00:36:47.566920 130972793685568 k8s_log_collector.py:71] [ns/psm-interop-client-20240409-0034-1pism] Starting log collection thread 130972793685568 for psm-grpc-client-6899fdb4c6-vzmlg
I0409 00:36:47.567100 130972906758144 k8s_base_runner.py:944] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0409 00:36:50.632166 130972906758144 k8s_base_runner.py:953] Deployment psm-grpc-client has 1 replicas available
I0409 00:36:50.632538 130972906758144 client_app.py:292] [psm-grpc-client-6899fdb4c6-vzmlg] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0409 00:36:50.674911 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:37:00.697016 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:37:12.729216 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:37:22.752586 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:37:32.776235 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:37:48.804134 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:38:13.839349 130972906758144 client_app.py:320] [psm-grpc-client-6899fdb4c6-vzmlg] xDS control plane channel:
I0409 00:38:15.846521 130972906758144 client_app.py:331] [psm-grpc-client-6899fdb4c6-vzmlg] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0409 00:38:15.846884 130972906758144 client_app.py:302] [psm-grpc-client-6899fdb4c6-vzmlg] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0409 00:38:15.847177 130972906758144 client_app.py:252] [psm-grpc-client-6899fdb4c6-vzmlg] Waiting to report a READY channel to xds:///psm-grpc-server:34030
I0409 00:38:15.851518 130972906758144 client_app.py:379] [psm-grpc-client-6899fdb4c6-vzmlg] Server channel:
I0409 00:38:15.860129 130972906758144 client_app.py:392] [psm-grpc-client-6899fdb4c6-vzmlg] Found subchannel in state READY:
I0409 00:38:15.860445 130972906758144 client_app.py:263] [psm-grpc-client-6899fdb4c6-vzmlg] Channel to xds:///psm-grpc-server:34030 transitioned to state READY:
I0409 00:38:15.860581 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.07_start_test_client ---
I0409 00:38:15.860666 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0409 00:38:15.860873 130972906758144 grpc.py:75] [psm-grpc-client-6899fdb4c6-vzmlg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=30)
I0409 00:38:15.876050 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0409 00:38:15.876205 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0409 00:38:15.876450 130972906758144 grpc.py:75] [psm-grpc-client-6899fdb4c6-vzmlg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:38:19.938805 130972906758144 xds_k8s_testcase.py:669] [psm-grpc-client-6899fdb4c6-vzmlg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-cf69bd678-sjm2h: 50
psm-grpc-server-cf69bd678-5gjv5: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-cf69bd678-sjm2h: 50
psm-grpc-server-cf69bd678-5gjv5: 50

I0409 00:38:19.939010 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0409 00:38:19.939092 130972906758144 xds_k8s_testcase.py:299] --- Starting subTest RoundRobinTest.test_round_robin.10_round_robin ---
I0409 00:38:19.939208 130972906758144 grpc.py:75] [psm-grpc-client-6899fdb4c6-vzmlg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0409 00:38:23.998790 130972906758144 xds_k8s_testcase.py:669] [psm-grpc-client-6899fdb4c6-vzmlg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-cf69bd678-sjm2h: 50
psm-grpc-server-cf69bd678-5gjv5: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-cf69bd678-sjm2h: 50
psm-grpc-server-cf69bd678-5gjv5: 50

I0409 00:38:23.999137 130972906758144 xds_k8s_testcase.py:304] --- Finished subTest RoundRobinTest.test_round_robin.10_round_robin ---
I0409 00:38:23.999230 130972906758144 xds_k8s_testcase.py:755] ----- TestMethod RoundRobinTest.test_round_robin teardown -----
I0409 00:38:24.042665 130972906758144 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240409-0034-1pism"
I0409 00:38:24.671988 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712623104050-6159f23c0471a-167386f2-ad35414f
I0409 00:38:37.744609 130972906758144 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240409-0034-1pism"
I0409 00:38:37.833193 130972906758144 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240409-0034-1pism"
I0409 00:38:38.288193 130972906758144 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240409-0034-1pism"
I0409 00:38:38.978737 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712623118290-6159f24999008-8fb6aba6-f1686cf0
I0409 00:38:41.273362 130972906758144 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240409-0034-1pism"
I0409 00:38:41.606892 130972906758144 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240409-0034-1pism"
I0409 00:38:42.310580 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712623121611-6159f24cc3c09-a39d49b8-65478911
I0409 00:38:44.629734 130972906758144 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240409-0034-1pism"
I0409 00:38:45.123076 130972906758144 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240409-0034-1pism"
I0409 00:38:45.669434 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712623125132-6159f2501f501-f735fe5e-a8d57cd7
I0409 00:39:07.344115 130972906758144 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240409-0034-1pism"
I0409 00:39:07.765443 130972906758144 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240409-0034-1pism"
I0409 00:39:08.204025 130972906758144 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240409-0034-1pism"
I0409 00:39:08.699305 130972906758144 compute.py:614] Waiting 600 sec for compute operation id: operation-1712623148278-6159f266321ab-2916f86d-62af11e0
I0409 00:39:13.173116 130972906758144 k8s_base_runner.py:873] Deleting deployment psm-grpc-client
I0409 00:39:53.329173 130972906758144 k8s_base_runner.py:888] Deployment psm-grpc-client deleted
I0409 00:39:53.329449 130972906758144 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240409-0034-1pism/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:39:53.689550 130972906758144 k8s_base_runner.py:907] Deleting service account psm-grpc-client
I0409 00:39:53.737569 130972906758144 k8s_base_runner.py:921] Service account psm-grpc-client deleted
I0409 00:39:53.737842 130972906758144 k8s_base_runner.py:924] Deleting namespace psm-interop-client-20240409-0034-1pism
I0409 00:40:23.823457 130972906758144 k8s_base_runner.py:941] Namespace psm-interop-client-20240409-0034-1pism deleted
I0409 00:40:23.823685 130972906758144 k8s_base_runner.py:873] Deleting deployment psm-grpc-server
I0409 00:40:33.903584 130972906758144 k8s_base_runner.py:888] Deployment psm-grpc-server deleted
I0409 00:40:33.903813 130972906758144 k8s_base_runner.py:891] Deleting service psm-grpc-server
I0409 00:40:34.958390 130972906758144 k8s_base_runner.py:904] Service psm-grpc-server deleted
I0409 00:40:34.958618 130972906758144 k8s_base_runner.py:471] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240409-0034-1pism/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0409 00:40:35.444535 130972906758144 k8s_base_runner.py:907] Deleting service account psm-grpc-server
I0409 00:40:35.497250 130972906758144 k8s_base_runner.py:921] Service account psm-grpc-server deleted
I0409 00:40:35.497440 130972906758144 k8s_base_runner.py:924] Deleting namespace psm-interop-server-20240409-0034-1pism
I0409 00:41:05.580516 130972906758144 k8s_base_runner.py:941] Namespace psm-interop-server-20240409-0034-1pism deleted
I0409 00:41:05.580834 130972906758144 xds_k8s_testcase.py:779] ----- Test client/server logs -----
I0409 00:41:05.581047 130972906758144 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-client-dmrvslxa:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240409-0034-1pism%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-dmrvslxa%22;timeRange=2024-04-09T00:36:42.838760Z%2F2024-04-09T00:40:23.823655Z;cursorTimestamp=2024-04-09T00:36:50.632379Z?project=grpc-testing
I0409 00:41:05.581207 130972906758144 k8s_base_runner.py:1152] GCP Logs Explorer link to psm-grpc-server-1586e550:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240409-0034-1pism%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-1586e550%22;timeRange=2024-04-09T00:35:35.866161Z%2F2024-04-09T00:41:05.580703Z;cursorTimestamp=2024-04-09T00:35:45.792818Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
I0409 00:41:05.581420 130972906758144 base_testcase.py:67] ----- PSM Test Case PASSED: RoundRobinTest.test_round_robin -----

----------------------------------------------------------------------
Ran 1 test in 379.659s

OK
Failed test suites: 0


[ID: 4869297] Command finished after 4884 secs, exit value: 0


Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[17:41:06 PDT] Collecting build artifacts from build VM
[17:41:11 PDT] Kokoro builder finished