Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[09:59:00 PST] Transferring environment variable script to build VM
[09:59:01 PST] Transferring kokoro_log_reader.py to build VM
[09:59:01 PST] Transferring source code to build VM
[09:59:02 PST] Executing build script on build VM



[ID: 6142015] Executing command via SSH:
export KOKORO_BUILD_NUMBER="703"
export KOKORO_JOB_NAME="grpc/go/master/branch/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-go/test/kokoro/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-go/test/kokoro/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-go
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/zasweq/grpc-go.git
GIT_COMMIT_SHORT,eb521e50
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 130042 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129987 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129587 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_0%3a459.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../kubectl_1%3a459.0.0-0_amd64.deb ...
Unpacking kubectl (1:459.0.0-0) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../python3-pip-whl_22.0.2+dfsg-1ubuntu0.4_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../python3.10-venv_3.10.12-1~22.04.3_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.3) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Setting up kubectl (1:459.0.0-0) ...
Setting up python3.10-venv (3.10.12-1~22.04.3) ...
Processing triggers for man-db (2.10.2-1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.6oxhUDkPHo/psm-interop
Cloning into '/tmp/tmp.6oxhUDkPHo/psm-interop'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.6oxhUDkPHo/psm-interop/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 2.4 MB/s eta 0:00:00
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 17.5 MB/s eta 0:00:00
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 13.7 MB/s eta 0:00:00
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 5.3 MB/s eta 0:00:00
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 8.5 MB/s eta 0:00:00
Collecting grpcio==1.57.0 (from -r requirements.lock (line 6))
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting grpcio-health-checking==1.57.0 (from -r requirements.lock (line 7))
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting grpcio-tools==1.57.0 (from -r requirements.lock (line 8))
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio-channelz==1.57.0 (from -r requirements.lock (line 9))
Downloading grpcio_channelz-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 10))
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 11))
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 12))
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==23.1 (from -r requirements.lock (line 13))
Downloading packaging-23.1-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 4.2 MB/s eta 0:00:00
Collecting Pygments==2.14.0 (from -r requirements.lock (line 14))
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 46.4 MB/s eta 0:00:00
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 15))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 19.3 MB/s eta 0:00:00
Collecting protobuf==4.24.1 (from -r requirements.lock (line 16))
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting xds-protos==1.58.0rc1 (from -r requirements.lock (line 17))
Downloading xds_protos-1.58.0rc1-py3-none-any.whl.metadata (1.3 kB)
Collecting cachetools==5.3.1 (from -r requirements.lock (line 19))
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.lock (line 20))
Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.2.0 (from -r requirements.lock (line 21))
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting google-api-core==2.11.1 (from -r requirements.lock (line 22))
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0 (from -r requirements.lock (line 23))
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 24))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.60.0 (from -r requirements.lock (line 25))
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 26))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.57.0 (from -r requirements.lock (line 27))
Downloading grpcio_status-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting httplib2==0.22.0 (from -r requirements.lock (line 28))
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 8.5 MB/s eta 0:00:00
Collecting idna==3.4 (from -r requirements.lock (line 29))
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 5.1 MB/s eta 0:00:00
Collecting MarkupSafe==2.1.3 (from -r requirements.lock (line 30))
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 31))
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 14.4 MB/s eta 0:00:00
Collecting proto-plus==1.22.3 (from -r requirements.lock (line 32))
Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.lock (line 33))
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 7.8 MB/s eta 0:00:00
Collecting pyasn1-modules==0.3.0 (from -r requirements.lock (line 34))
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 10.1 MB/s eta 0:00:00
Collecting pyparsing==3.1.1 (from -r requirements.lock (line 35))
Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting requests==2.31.0 (from -r requirements.lock (line 36))
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 37))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 38))
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 39))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.16 (from -r requirements.lock (line 40))
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 4.1 MB/s eta 0:00:00
Collecting websocket-client==1.6.2 (from -r requirements.lock (line 41))
Downloading websocket_client-1.6.2-py3-none-any.whl.metadata (7.5 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.57.0->-r requirements.lock (line 8)) (69.0.3)
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 93.2 MB/s eta 0:00:00
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl (8.6 kB)
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 83.3 MB/s eta 0:00:00
Downloading grpcio_channelz-1.57.0-py3-none-any.whl (10 kB)
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 63.1 MB/s eta 0:00:00
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.4/311.4 kB 22.2 MB/s eta 0:00:00
Downloading xds_protos-1.58.0rc1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 50.2 MB/s eta 0:00:00
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 12.8 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.8/201.8 kB 16.4 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 10.0 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 15.8 MB/s eta 0:00:00
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 19.2 MB/s eta 0:00:00
Downloading grpcio_status-1.57.0-py3-none-any.whl (5.1 kB)
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 4.3 MB/s eta 0:00:00
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 8.8 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 5.2 MB/s eta 0:00:00
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 12.8 MB/s eta 0:00:00
Downloading websocket_client-1.6.2-py3-none-any.whl (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 4.8 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, uritemplate, six, PyYAML, pyparsing, Pygments, pyasn1, protobuf, packaging, oauthlib, MarkupSafe, idna, grpcio, charset-normalizer, certifi, cachetools, xds-protos, tenacity, rsa, requests, python-dateutil, pyasn1-modules, proto-plus, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, absl-py, requests-oauthlib, grpcio-status, google-auth, kubernetes, grpc-google-iam-v1, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-secret-manager
Successfully installed Mako-1.2.4 MarkupSafe-2.1.3 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-api-core-2.11.1 google-api-python-client-1.12.11 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.57.0 grpcio-channelz-1.57.0 grpcio-health-checking-1.57.0 grpcio-status-1.57.0 grpcio-tools-1.57.0 httplib2-0.22.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.1 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.16 websocket-client-1.6.2 xds-protos-1.58.0rc1
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.1
certifi 2023.7.22
charset-normalizer 3.2.0
google-api-core 2.11.1
google-api-python-client 1.12.11
google-auth 2.22.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.60.0
grpc-google-iam-v1 0.12.6
grpcio 1.57.0
grpcio-channelz 1.57.0
grpcio-health-checking 1.57.0
grpcio-status 1.57.0
grpcio-tools 1.57.0
httplib2 0.22.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.3
oauthlib 3.2.2
packaging 23.1
pip 23.3.2
proto-plus 1.22.3
protobuf 4.24.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
Pygments 2.14.0
pyparsing 3.1.1
python-dateutil 2.8.2
PyYAML 6.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.0.3
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.16
websocket-client 1.6.2
xds-protos 1.58.0rc1
Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto
/tmp/tmp.6oxhUDkPHo/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.6oxhUDkPHo/psm-interop/protos/grpc/testing:
total 108
-rw-rw-r-- 1 kbuilder kbuilder 957 Jan 16 17:59 empty.proto
-rw-rw-r-- 1 kbuilder kbuilder 1007 Jan 16 18:00 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 16 18:00 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 11527 Jan 16 17:59 messages.proto
-rw-rw-r-- 1 kbuilder kbuilder 17206 Jan 16 18:00 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 16 18:00 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 4871 Jan 16 17:59 test.proto
-rw-rw-r-- 1 kbuilder kbuilder 4296 Jan 16 18:00 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 41870 Jan 16 18:00 test_pb2_grpc.py
Server image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
Server image not found
Client image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
Client image not found
Building Go xDS interop test app Docker images
#0 building with "default" instance using docker driver

#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.51kB done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 ...

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 2.1s

#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 3.3s

#5 [internal] load build context
#5 DONE 0.0s

#6 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#6 resolve docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 0.0s done
#6 sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 1.64kB / 1.64kB done
#6 sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd 528B / 528B done
#6 sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418 1.47kB / 1.47kB done
#6 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0B / 3.41MB 0.1s
#6 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.2s done
#6 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca
#6 DONE 0.6s

#7 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#7 resolve docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 0.1s done
#7 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 37.14MB / 67.09MB 0.5s
#7 sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 1.65kB / 1.65kB done
#7 sha256:2523a6f68a0f515fe251aad40b18545155135ca6a5b2e61da8254df9153e3648 1.16kB / 1.16kB done
#7 sha256:82e66f0646744e5c40c91b75ff7d5ff5577887b46d5402965633324dd842ea87 5.22kB / 5.22kB done
#7 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.2s done
#7 sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 284.20kB / 284.20kB 0.2s done
#7 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0.2s done
#7 sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 156B / 156B 0.3s done
#7 extracting sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d
#7 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 42.99MB / 67.09MB 0.6s
#7 extracting sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 0.1s done
#7 ...

#5 [internal] load build context
#5 transferring context: 87.14MB 0.7s done
#5 DONE 0.7s

#7 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#7 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 60.82MB / 67.09MB 0.8s
#7 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 67.09MB / 67.09MB 1.0s done
#7 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 0.1s
#7 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 5.2s
#7 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 6.3s done
#7 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98
#7 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 done
#7 DONE 7.8s

#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 DONE 1.2s

#9 [build 3/4] COPY . .
#9 DONE 0.5s

#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/client/client.go
#10 0.373 go: downloading github.com/golang/protobuf v1.5.3
#10 0.377 go: downloading google.golang.org/protobuf v1.31.0
#10 0.396 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.442 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.456 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.494 go: downloading golang.org/x/sys v0.14.0
#10 0.504 go: downloading golang.org/x/net v0.18.0
#10 0.755 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.869 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 1.192 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.202 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.202 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.205 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.209 go: downloading golang.org/x/text v0.14.0
#10 1.210 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.212 go: downloading golang.org/x/sync v0.5.0
#10 1.266 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.275 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 16.2s

#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/client .
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers 0.1s done
#12 writing image sha256:fafb1f043a7993b053b7c67939a3b39ced3b1c9ce809bfb6fcf19ae3138bcfb9 done
#12 naming to gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 done
#12 DONE 0.2s
#0 building with "default" instance using docker driver

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.51kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.1s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.2s

#5 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#5 DONE 0.0s

#6 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#6 CACHED

#7 [internal] load build context
#7 transferring context: 76.35kB 0.1s done
#7 DONE 0.1s

#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 CACHED

#9 [build 3/4] COPY . .
#9 CACHED

#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/server/server.go
#10 0.355 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.367 go: downloading github.com/golang/protobuf v1.5.3
#10 0.367 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.367 go: downloading google.golang.org/protobuf v1.31.0
#10 0.440 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.508 go: downloading golang.org/x/net v0.18.0
#10 0.753 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.806 go: downloading golang.org/x/sys v0.14.0
#10 0.872 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.897 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.232 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.232 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.242 go: downloading golang.org/x/text v0.14.0
#10 1.247 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.248 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.251 go: downloading golang.org/x/sync v0.5.0
#10 1.324 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.335 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 16.0s

#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/server .
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:0b2918e4e9a7bcb560591a529871efbe1a23db09a5718ee23d98676c70b14244 done
#12 naming to gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 done
#12 DONE 0.2s
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-client]
86879a8abfb3: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
86879a8abfb3: Pushed
eb521e502093ca03aea6f558edf5fa36a7caf723: digest: sha256:94b8972842fc358619f10eb230659cce39f5ead1e2d64c3b769b797294964465 size: 740
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-server]
3d99b714767e: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
3d99b714767e: Pushed
eb521e502093ca03aea6f558edf5fa36a7caf723: digest: sha256:1d1ff562457e95dc3783be41cbd30dc0daf8367139cec5dbf3be103af4d23d56 size: 740
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test'
+ python -m tests.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:01:08.713656 140280860045312 xds_k8s_testcase.py:157] ----- Testing AffinityTest -----
I0116 18:01:08.713775 140280860045312 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:01:08.713960 140280860045312 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:01:08.728196 140280860045312 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:01:08.884409 140280860045312 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0116 18:01:09.160909 140280860045312 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1801-a7r7r
I0116 18:01:09.395389 140280860045312 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test
I0116 18:01:09.626125 140280860045312 xds_k8s_testcase.py:663] Found unused xds port: 26447
I0116 18:01:09.626607 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0116 18:01:09.626744 140280860045312 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1801-a7r7r"
I0116 18:01:09.629312 140280860045312 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1801-a7r7r
type: GRPC
...

I0116 18:01:10.268813 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428069685-60f13ea48986d-23dcaf8d-ab47a284
I0116 18:01:12.576763 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0116 18:01:12.577009 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0116 18:01:12.577134 140280860045312 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1801-a7r7r"
I0116 18:01:12.583323 140280860045312 compute.py:537] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1801-a7r7r
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20240116-1801-a7r7r
protocol: GRPC
sessionAffinity: HEADER_FIELD
...

I0116 18:01:13.368023 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428072585-60f13ea74d9ec-409bbcc6-9eee9a44
I0116 18:01:28.409980 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0116 18:01:28.410240 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0116 18:01:28.410353 140280860045312 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1801-a7r7r": psm-grpc-server:26447 -> psm-interop-backend-service-20240116-1801-a7r7r
I0116 18:01:28.418658 140280860045312 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1801-a7r7r
hostRules:
- hosts:
- psm-grpc-server:26447
pathMatcher: psm-interop-path-matcher-20240116-1801-a7r7r
name: psm-interop-url-map-20240116-1801-a7r7r
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1801-a7r7r
name: psm-interop-path-matcher-20240116-1801-a7r7r
...

I0116 18:01:29.152037 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428088421-60f13eb667af3-72c68724-d5d7af7c
I0116 18:01:31.392818 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0116 18:01:31.393066 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0116 18:01:31.393181 140280860045312 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1801-a7r7r proxy "GRPC" to URL map psm-interop-url-map-20240116-1801-a7r7r
I0116 18:01:31.394859 140280860045312 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1801-a7r7r
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1801-a7r7r
validate_for_proxyless: true
...

I0116 18:01:32.014682 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428091396-60f13eb93e274-00c627b1-699a5a7d
I0116 18:01:34.264681 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0116 18:01:34.264916 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0116 18:01:34.265044 140280860045312 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1801-a7r7r" in network "default-vpc": 0.0.0.0:26447 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1801-a7r7r
I0116 18:01:34.266967 140280860045312 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1801-a7r7r
network: global/networks/default-vpc
portRange: 26447
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1801-a7r7r
...

I0116 18:01:35.062227 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428094269-60f13ebbfb887-4d845b62-5e9d8235
I0116 18:01:48.062563 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0116 18:01:48.062835 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0116 18:01:48.062978 140280860045312 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1801-a7r7r: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0116 18:01:48.064897 140280860045312 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1801-a7r7r%22;timeRange=2024-01-16T18:01:48.063078Z%2F2024-01-16T18:31:48.063078Z?project=grpc-testing
I0116 18:01:48.069015 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1801-a7r7r
labels:
name: psm-interop-server-20240116-1801-a7r7r
owner: xds-k8s-interop-test
...

I0116 18:01:48.143540 140280860045312 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1801-a7r7r created
I0116 18:01:48.146055 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1801-a7r7r
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1801-a7r7r-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:01:48.213703 140280860045312 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:01:48.213932 140280860045312 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1801-a7r7r
I0116 18:01:48.308418 140280860045312 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1801-a7r7r: neg_name=psm-interop-server-20240116-1801-a7r7r-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:01:48.308693 140280860045312 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1801-a7r7r/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:01:49.011269 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1801-a7r7r
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:01:49.067559 140280860045312 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:01:49.076060 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1801-a7r7r
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-3z783wlc
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-3z783wlc
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-3z783wlc
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1801-a7r7r-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:01:49.146147 140280860045312 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:01:49.146382 140280860045312 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0116 18:01:49.266091 140280860045312 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-5f85bc8df7-4pb9b', 'psm-grpc-server-5f85bc8df7-n57k4', 'psm-grpc-server-5f85bc8df7-w7xn2']
I0116 18:01:49.266484 140280860045312 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5f85bc8df7-4pb9b to start
I0116 18:01:52.559964 140280860045312 k8s_base_runner.py:820] Pod psm-grpc-server-5f85bc8df7-4pb9b ready, IP: 10.120.4.143
I0116 18:01:52.560260 140280860045312 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5f85bc8df7-4pb9b to affinity_test/test_app_logs/psm-interop-server-20240116-1801-a7r7r_psm-grpc-server-5f85bc8df7-4pb9b.log
I0116 18:01:52.560931 140280766916160 k8s_log_collector.py:70] Starting log collection thread 140280766916160 for psm-grpc-server-5f85bc8df7-4pb9b
I0116 18:01:52.561163 140280860045312 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5f85bc8df7-n57k4 to start
I0116 18:01:52.656532 140280860045312 k8s_base_runner.py:820] Pod psm-grpc-server-5f85bc8df7-n57k4 ready, IP: 10.120.0.27
I0116 18:01:52.656812 140280860045312 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5f85bc8df7-n57k4 to affinity_test/test_app_logs/psm-interop-server-20240116-1801-a7r7r_psm-grpc-server-5f85bc8df7-n57k4.log
I0116 18:01:52.657437 140280758523456 k8s_log_collector.py:70] Starting log collection thread 140280758523456 for psm-grpc-server-5f85bc8df7-n57k4
I0116 18:01:52.657627 140280860045312 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5f85bc8df7-w7xn2 to start
I0116 18:01:52.828410 140280860045312 k8s_base_runner.py:820] Pod psm-grpc-server-5f85bc8df7-w7xn2 ready, IP: 10.120.3.66
I0116 18:01:52.828669 140280860045312 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5f85bc8df7-w7xn2 to affinity_test/test_app_logs/psm-interop-server-20240116-1801-a7r7r_psm-grpc-server-5f85bc8df7-w7xn2.log
I0116 18:01:52.829286 140280678315584 k8s_log_collector.py:70] Starting log collection thread 140280678315584 for psm-grpc-server-5f85bc8df7-w7xn2
I0116 18:01:52.829504 140280860045312 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0116 18:01:58.197174 140280860045312 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0116 18:01:58.197795 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0116 18:01:58.197935 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0116 18:01:58.234605 140280860045312 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:01:58.523695 140280860045312 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1801-a7r7r-psm-grpc-server" in zone us-central1-a
I0116 18:01:58.523935 140280860045312 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1801-a7r7r: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1801-a7r7r-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1801-a7r7r-psm-grpc-server', zone='us-central1-a')}
I0116 18:01:58.527176 140280860045312 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1801-a7r7r-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 18:01:59.604244 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428118529-60f13ed31e4c8-7c0f0ea7-df7d2047
I0116 18:02:48.977007 140280860045312 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1801-a7r7r to report all backends healthy: ['psm-interop-server-20240116-1801-a7r7r-psm-grpc-server']
I0116 18:03:01.681149 140280860045312 compute.py:494] Backend psm-interop-server-20240116-1801-a7r7r-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:03:05.686177 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0116 18:03:05.686459 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0116 18:03:05.686699 140280860045312 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1801-a7r7r: server_target=xds:///psm-grpc-server:26447 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0116 18:03:05.686960 140280860045312 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1801-a7r7r%22;timeRange=2024-01-16T18:03:05.686814Z%2F2024-01-16T18:33:05.686814Z?project=grpc-testing
I0116 18:03:05.688849 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1801-a7r7r
labels:
name: psm-interop-client-20240116-1801-a7r7r
owner: xds-k8s-interop-test
...

I0116 18:03:05.730352 140280860045312 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1801-a7r7r created
I0116 18:03:05.730609 140280860045312 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1801-a7r7r/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:03:07.744843 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1801-a7r7r
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:03:07.802247 140280860045312 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:03:07.809269 140280860045312 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1801-a7r7r
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-z4rrk4lw
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-z4rrk4lw
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-z4rrk4lw
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26447"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:03:07.875576 140280860045312 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:03:07.875818 140280860045312 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:03:07.975843 140280860045312 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-f675b4d67-hm8hg']
I0116 18:03:07.976146 140280860045312 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-f675b4d67-hm8hg to start
I0116 18:03:11.207517 140280860045312 k8s_base_runner.py:820] Pod psm-grpc-client-f675b4d67-hm8hg ready, IP: 10.120.7.31
I0116 18:03:11.207854 140280860045312 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-f675b4d67-hm8hg to affinity_test/test_app_logs/psm-interop-client-20240116-1801-a7r7r_psm-grpc-client-f675b4d67-hm8hg.log
I0116 18:03:11.208567 140280669922880 k8s_log_collector.py:70] Starting log collection thread 140280669922880 for psm-grpc-client-f675b4d67-hm8hg
I0116 18:03:11.208842 140280860045312 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:03:12.419025 140280860045312 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:03:12.419589 140280860045312 client_app.py:284] [psm-grpc-client-f675b4d67-hm8hg] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:03:12.523332 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:03:24.602608 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:03:34.647433 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:03:44.691883 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:03:54.728672 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:04:10.777887 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:04:35.837428 140280860045312 client_app.py:312] [psm-grpc-client-f675b4d67-hm8hg] xDS control plane channel:
I0116 18:04:37.872548 140280860045312 client_app.py:323] [psm-grpc-client-f675b4d67-hm8hg] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:04:37.872898 140280860045312 client_app.py:294] [psm-grpc-client-f675b4d67-hm8hg] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:04:37.873215 140280860045312 client_app.py:244] [psm-grpc-client-f675b4d67-hm8hg] Waiting to report a READY channel to xds:///psm-grpc-server:26447
I0116 18:04:37.905596 140280860045312 client_app.py:371] [psm-grpc-client-f675b4d67-hm8hg] Server channel:
I0116 18:04:38.001957 140280860045312 client_app.py:384] [psm-grpc-client-f675b4d67-hm8hg] Found subchannel in state READY:
I0116 18:04:38.002396 140280860045312 client_app.py:255] [psm-grpc-client-f675b4d67-hm8hg] Channel to xds:///psm-grpc-server:26447 transitioned to state READY:
I0116 18:04:38.002692 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:04:38.036804 140280860045312 affinity_test.py:108] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/87ab6ffc-2d0e-48d0-8f45-304500859e3d",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.7.31",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"versionInfo": "2",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7681754650773256802_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.27",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.66",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.4.143",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-16T18:04:35.107093817Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:26447",
"versionInfo": "1705428240633313497",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:26447",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-16T18:04:25.095516560Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447",
"versionInfo": "1705428240633313497",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:26447"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-16T18:04:25.098583670Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"versionInfo": "1705428240633313497",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240116-1801-a7r7r"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240116-1801-a7r7r",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-16T18:04:25.103408179Z",
"clientStatus": "ACKED"
}
]
}
I0116 18:04:38.037538 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0116 18:04:38.037673 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0116 18:04:38.037793 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:04:38.071188 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0116 18:04:38.071434 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0116 18:04:38.071706 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:04:42.081345 140280860045312 xds_k8s_testcase.py:601] [psm-grpc-client-f675b4d67-hm8hg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f85bc8df7-w7xn2: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5f85bc8df7-w7xn2: 100

I0116 18:04:42.081644 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0116 18:04:42.081768 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0116 18:04:42.081928 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:04:46.080982 140280860045312 xds_k8s_testcase.py:601] [psm-grpc-client-f675b4d67-hm8hg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f85bc8df7-w7xn2: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5f85bc8df7-w7xn2: 100

I0116 18:04:46.209693 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0116 18:04:46.209966 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0116 18:04:46.210070 140280860045312 server_app.py:98] [psm-grpc-server-5f85bc8df7-w7xn2] >> Setting health status to NOT_SERVING
I0116 18:04:46.210394 140280860045312 grpc.py:75] [psm-grpc-server-5f85bc8df7-w7xn2:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), wait_for_ready=True, timeout=90)
I0116 18:04:46.308370 140280860045312 grpc.py:75] [psm-grpc-server-5f85bc8df7-w7xn2:8080] >> RPC Health.Check(request=HealthCheckRequest({}), wait_for_ready=True, timeout=90)
I0116 18:04:46.340953 140280860045312 server_app.py:102] [psm-grpc-server-5f85bc8df7-w7xn2] << Health status status: NOT_SERVING

I0116 18:04:46.341221 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0116 18:04:46.341327 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0116 18:04:46.341457 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:04:46.374657 140280860045312 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0116 18:04:56.385140 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:04:56.419982 140280860045312 affinity_test.py:185] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/87ab6ffc-2d0e-48d0-8f45-304500859e3d",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.7.31",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:26447",
"versionInfo": "1705428273232152332",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:26447",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-16T18:04:40.162491615Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447",
"versionInfo": "1705428273232152332",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:26447"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1801-a7r7r_0_psm-grpc-server:26447-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-16T18:04:40.162886434Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"versionInfo": "1705428273232152332",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240116-1801-a7r7r"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240116-1801-a7r7r",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-16T18:04:40.162228905Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"versionInfo": "4",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_5276216937994222214",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7681754650773256802_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.27",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.66",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.4.143",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-16T18:04:55.106666522Z",
"clientStatus": "ACKED"
}
]
}
I0116 18:04:56.420625 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0116 18:04:56.420754 140280860045312 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0116 18:04:56.420950 140280860045312 grpc.py:75] [psm-grpc-client-f675b4d67-hm8hg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:05:00.441223 140280860045312 xds_k8s_testcase.py:601] [psm-grpc-client-f675b4d67-hm8hg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f85bc8df7-n57k4: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5f85bc8df7-n57k4: 100

I0116 18:05:00.441553 140280860045312 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0116 18:05:00.441736 140280860045312 xds_k8s_testcase.py:687] ----- TestMethod __main__.AffinityTest.test_affinity teardown -----
I0116 18:05:00.684358 140280860045312 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1801-a7r7r"
I0116 18:05:01.419588 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428300688-60f13f80d6a80-1c6b017a-51f7de88
I0116 18:05:14.434215 140280860045312 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1801-a7r7r"
I0116 18:05:14.536501 140280860045312 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1801-a7r7r"
I0116 18:05:15.009819 140280860045312 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1801-a7r7r"
I0116 18:05:15.493335 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428315013-60f13f8e7ffa9-2665ad41-c0aed3cd
I0116 18:05:17.771749 140280860045312 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1801-a7r7r"
I0116 18:05:18.077230 140280860045312 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1801-a7r7r"
I0116 18:05:18.859379 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428318081-60f13f916d385-b7716ffa-b3acf2bf
I0116 18:05:21.120203 140280860045312 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1801-a7r7r"
I0116 18:05:21.594713 140280860045312 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1801-a7r7r"
I0116 18:05:22.195986 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428321599-60f13f94c81e4-c890220d-09c07dd8
I0116 18:05:48.003364 140280860045312 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1801-a7r7r"
I0116 18:05:48.355561 140280860045312 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1801-a7r7r"
I0116 18:05:48.790170 140280860045312 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1801-a7r7r"
I0116 18:05:49.300111 140280860045312 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428348855-60f13faec663d-87647ad4-eec9f663
I0116 18:05:53.658613 140280860045312 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:06:03.825957 140280860045312 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:06:03.826337 140280860045312 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1801-a7r7r/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:06:04.409482 140280860045312 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:06:04.523951 140280860045312 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:06:04.524224 140280860045312 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1801-a7r7r
I0116 18:06:34.706392 140280860045312 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1801-a7r7r deleted
I0116 18:06:34.706732 140280860045312 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:06:44.873834 140280860045312 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:06:44.874179 140280860045312 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:06:44.988525 140280860045312 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:06:44.988819 140280860045312 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1801-a7r7r/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:06:45.586920 140280860045312 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:06:45.703196 140280860045312 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:06:45.703460 140280860045312 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1801-a7r7r
I0116 18:08:46.093482 140280860045312 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1801-a7r7r deleted
I0116 18:08:46.093851 140280860045312 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:08:46.094074 140280860045312 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-z4rrk4lw:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1801-a7r7r%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-z4rrk4lw%22;timeRange=2024-01-16T18:03:05.686805Z%2F2024-01-16T18:06:34.706697Z?project=grpc-testing
I0116 18:08:46.094197 140280860045312 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-3z783wlc:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1801-a7r7r%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-3z783wlc%22;timeRange=2024-01-16T18:01:48.063069Z%2F2024-01-16T18:08:46.093661Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
I0116 18:08:46.094390 140280860045312 base_testcase.py:52] ----- TestCase __main__.AffinityTest.test_affinity PASSED -----
----------------------------------------------------------------------
Ran 1 test in 457.383s

OK
+ for test in "${test_suites[@]}"
+ run_test api_listener_test
+ local test_name=api_listener_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test'
+ set -x
+ python -m tests.api_listener_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:08:47.392171 140090947739648 xds_k8s_testcase.py:157] ----- Testing ApiListenerTest -----
I0116 18:08:47.392314 140090947739648 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:08:47.392620 140090947739648 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:08:47.407361 140090947739648 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:08:47.422155 140090947739648 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0116 18:08:47.426563 140090947739648 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1808-1ajrg
I0116 18:08:47.685543 140090947739648 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
I0116 18:08:47.999691 140090947739648 xds_k8s_testcase.py:663] Found unused xds port: 43176
I0116 18:08:48.000144 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0116 18:08:48.000295 140090947739648 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1808-1ajrg"
I0116 18:08:48.002760 140090947739648 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1808-1ajrg
type: GRPC
...

I0116 18:08:48.716625 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428528062-60f14059ae20c-0e5eb95b-4df42399
I0116 18:08:51.085855 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0116 18:08:51.086116 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0116 18:08:51.086264 140090947739648 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1808-1ajrg"
I0116 18:08:51.092167 140090947739648 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1808-1ajrg
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1808-1ajrg
protocol: GRPC
...

I0116 18:08:51.803400 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428531094-60f1405c922b2-3c0fbcc0-afe8627c
I0116 18:09:19.669025 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0116 18:09:19.669290 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0116 18:09:19.669416 140090947739648 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1808-1ajrg": psm-grpc-server:43176 -> psm-interop-backend-service-20240116-1808-1ajrg
I0116 18:09:19.678829 140090947739648 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1808-1ajrg
hostRules:
- hosts:
- psm-grpc-server:43176
pathMatcher: psm-interop-path-matcher-20240116-1808-1ajrg
name: psm-interop-url-map-20240116-1808-1ajrg
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1808-1ajrg
name: psm-interop-path-matcher-20240116-1808-1ajrg
...

I0116 18:09:20.333860 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428559681-60f14077d566b-6e484989-178c0310
I0116 18:09:22.659359 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0116 18:09:22.659606 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0116 18:09:22.659745 140090947739648 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1808-1ajrg proxy "GRPC" to URL map psm-interop-url-map-20240116-1808-1ajrg
I0116 18:09:22.661571 140090947739648 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1808-1ajrg
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1808-1ajrg
validate_for_proxyless: true
...

I0116 18:09:23.249535 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428562663-60f1407aad8f5-cfc07507-49710efc
I0116 18:09:25.490300 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0116 18:09:25.490575 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0116 18:09:25.490781 140090947739648 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1808-1ajrg" in network "default-vpc": 0.0.0.0:43176 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1808-1ajrg
I0116 18:09:25.493060 140090947739648 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1808-1ajrg
network: global/networks/default-vpc
portRange: 43176
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1808-1ajrg
...

I0116 18:09:26.277921 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428565494-60f1407d60c3a-ae5b84d3-1e762654
I0116 18:10:15.022075 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0116 18:10:15.022338 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0116 18:10:15.022473 140090947739648 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1808-1ajrg: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 18:10:15.022763 140090947739648 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1808-1ajrg%22;timeRange=2024-01-16T18:10:15.022611Z%2F2024-01-16T18:40:15.022611Z?project=grpc-testing
I0116 18:10:15.027011 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1808-1ajrg
labels:
name: psm-interop-server-20240116-1808-1ajrg
owner: xds-k8s-interop-test
...

I0116 18:10:15.180714 140090947739648 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1808-1ajrg created
I0116 18:10:15.183139 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1808-1ajrg
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1808-1ajrg-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:10:15.248347 140090947739648 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:10:15.248654 140090947739648 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1808-1ajrg
I0116 18:10:15.340943 140090947739648 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1808-1ajrg: neg_name=psm-interop-server-20240116-1808-1ajrg-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:10:15.341169 140090947739648 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1808-1ajrg/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:10:15.987560 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1808-1ajrg
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:10:16.047396 140090947739648 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:10:16.055122 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1808-1ajrg
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2bzz5d6d
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2bzz5d6d
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-2bzz5d6d
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1808-1ajrg-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:10:16.120841 140090947739648 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:10:16.121105 140090947739648 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 18:10:16.228058 140090947739648 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-8d8f67d9d-h659f']
I0116 18:10:16.228363 140090947739648 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-8d8f67d9d-h659f to start
I0116 18:10:19.466223 140090947739648 k8s_base_runner.py:820] Pod psm-grpc-server-8d8f67d9d-h659f ready, IP: 10.120.5.126
I0116 18:10:19.466505 140090947739648 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-8d8f67d9d-h659f to api_listener_test/test_app_logs/psm-interop-server-20240116-1808-1ajrg_psm-grpc-server-8d8f67d9d-h659f.log
I0116 18:10:19.467121 140090854073920 k8s_log_collector.py:70] Starting log collection thread 140090854073920 for psm-grpc-server-8d8f67d9d-h659f
I0116 18:10:19.469054 140090947739648 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 18:10:26.927750 140090947739648 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 18:10:26.928212 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0116 18:10:26.928343 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0116 18:10:26.965363 140090947739648 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:10:27.082270 140090947739648 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1808-1ajrg-psm-grpc-server" in zone us-central1-a
I0116 18:10:27.082525 140090947739648 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1808-1ajrg: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1808-1ajrg-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1808-1ajrg-psm-grpc-server', zone='us-central1-a')}
I0116 18:10:27.086222 140090947739648 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1808-1ajrg-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 18:10:28.159181 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428627088-60f140b81e49a-006b9a23-4ec0a767
I0116 18:11:13.287758 140090947739648 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1808-1ajrg to report all backends healthy: ['psm-interop-server-20240116-1808-1ajrg-psm-grpc-server']
I0116 18:11:17.673081 140090947739648 compute.py:494] Backend psm-interop-server-20240116-1808-1ajrg-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:11:21.677996 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0116 18:11:21.678307 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0116 18:11:21.678511 140090947739648 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1808-1ajrg: server_target=xds:///psm-grpc-server:43176 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 18:11:21.679078 140090947739648 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1808-1ajrg%22;timeRange=2024-01-16T18:11:21.678667Z%2F2024-01-16T18:41:21.678667Z?project=grpc-testing
I0116 18:11:21.681285 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1808-1ajrg
labels:
name: psm-interop-client-20240116-1808-1ajrg
owner: xds-k8s-interop-test
...

I0116 18:11:21.724970 140090947739648 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1808-1ajrg created
I0116 18:11:21.725238 140090947739648 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1808-1ajrg/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:11:22.318543 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1808-1ajrg
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:11:22.375815 140090947739648 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:11:22.382634 140090947739648 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1808-1ajrg
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-107cznmy
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-107cznmy
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-107cznmy
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:43176"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:11:22.516227 140090947739648 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:11:22.516532 140090947739648 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:11:22.614242 140090947739648 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-678cc7585b-8kks7']
I0116 18:11:22.614515 140090947739648 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-678cc7585b-8kks7 to start
I0116 18:11:25.847937 140090947739648 k8s_base_runner.py:820] Pod psm-grpc-client-678cc7585b-8kks7 ready, IP: 10.120.7.32
I0116 18:11:25.848232 140090947739648 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-678cc7585b-8kks7 to api_listener_test/test_app_logs/psm-interop-client-20240116-1808-1ajrg_psm-grpc-client-678cc7585b-8kks7.log
I0116 18:11:25.848868 140090845681216 k8s_log_collector.py:70] Starting log collection thread 140090845681216 for psm-grpc-client-678cc7585b-8kks7
I0116 18:11:25.849355 140090947739648 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:11:27.052346 140090947739648 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:11:27.052968 140090947739648 client_app.py:284] [psm-grpc-client-678cc7585b-8kks7] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:11:27.158520 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:11:37.203939 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:11:47.248724 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:11:57.293559 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:12:07.339925 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:12:23.389670 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:12:48.449973 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:13:13.510336 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:13:38.570170 140090947739648 client_app.py:312] [psm-grpc-client-678cc7585b-8kks7] xDS control plane channel:
I0116 18:13:40.605547 140090947739648 client_app.py:323] [psm-grpc-client-678cc7585b-8kks7] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:13:40.605908 140090947739648 client_app.py:294] [psm-grpc-client-678cc7585b-8kks7] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:13:40.606195 140090947739648 client_app.py:244] [psm-grpc-client-678cc7585b-8kks7] Waiting to report a READY channel to xds:///psm-grpc-server:43176
I0116 18:13:40.639052 140090947739648 client_app.py:371] [psm-grpc-client-678cc7585b-8kks7] Server channel:
I0116 18:13:40.672044 140090947739648 client_app.py:384] [psm-grpc-client-678cc7585b-8kks7] Found subchannel in state READY:
I0116 18:13:40.672574 140090947739648 client_app.py:255] [psm-grpc-client-678cc7585b-8kks7] Channel to xds:///psm-grpc-server:43176 transitioned to state READY:
I0116 18:13:40.672773 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0116 18:13:40.672873 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0116 18:13:40.673096 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 18:13:40.707852 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0116 18:13:40.708097 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0116 18:13:40.708336 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:13:44.726428 140090947739648 xds_k8s_testcase.py:601] [psm-grpc-client-678cc7585b-8kks7] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-8d8f67d9d-h659f: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-8d8f67d9d-h659f: 100

I0116 18:13:44.726706 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0116 18:13:44.726817 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0116 18:13:44.726920 140090947739648 traffic_director.py:540] Creating alternative URL map "psm-interop-url-map-alt-20240116-1808-1ajrg": psm-grpc-server:43176 -> psm-interop-backend-service-20240116-1808-1ajrg
I0116 18:13:44.730743 140090947739648 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1808-1ajrg
hostRules:
- hosts:
- psm-grpc-server:43176
pathMatcher: psm-interop-path-matcher-20240116-1808-1ajrg
name: psm-interop-url-map-alt-20240116-1808-1ajrg
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1808-1ajrg
name: psm-interop-path-matcher-20240116-1808-1ajrg
...

I0116 18:13:45.491965 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428824733-60f141749b7b1-e2250388-919717c9
I0116 18:13:47.774628 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0116 18:13:47.774888 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0116 18:13:47.774991 140090947739648 traffic_director.py:613] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20240116-1808-1ajrg" to URL map psm-interop-url-map-alt-20240116-1808-1ajrg
I0116 18:13:47.776732 140090947739648 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20240116-1808-1ajrg
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20240116-1808-1ajrg
validate_for_proxyless: false
...

I0116 18:13:48.243094 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428827778-60f1417782efd-b0c6830b-b692fb2a
I0116 18:13:50.464924 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0116 18:13:50.465175 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0116 18:13:50.465336 140090947739648 traffic_director.py:683] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20240116-1808-1ajrg" in network "default-vpc": 10.10.10.10:43176 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240116-1808-1ajrg
I0116 18:13:50.467580 140090947739648 compute.py:537] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20240116-1808-1ajrg
network: global/networks/default-vpc
portRange: 43176
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240116-1808-1ajrg
...

I0116 18:13:51.319138 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428830469-60f1417a13d11-9b9a08f4-9ef593c5
I0116 18:14:23.783381 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0116 18:14:23.783651 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0116 18:14:23.783860 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:14:27.805794 140090947739648 xds_k8s_testcase.py:601] [psm-grpc-client-678cc7585b-8kks7] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-8d8f67d9d-h659f: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-8d8f67d9d-h659f: 100

I0116 18:14:27.806100 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 18:14:27.839894 140090947739648 api_listener_test.py:110] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/aa74d31a-14a7-4a14-b884-98eb44d8ab18",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.7.32",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:43176",
"versionInfo": "1705428851436210466",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:43176",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240116-1808-1ajrg_0_psm-grpc-server:43176"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-16T18:14:22.248235821Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1808-1ajrg_0_psm-grpc-server:43176",
"versionInfo": "1705428851436210466",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1808-1ajrg_0_psm-grpc-server:43176",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:43176"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_612687301344721116",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240116-1808-1ajrg_0_psm-grpc-server:43176-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-16T18:14:22.248505031Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_612687301344721116",
"versionInfo": "1705428851436210466",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_612687301344721116",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240116-1808-1ajrg"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240116-1808-1ajrg",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-16T18:14:22.247825961Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_612687301344721116",
"versionInfo": "7",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_612687301344721116",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_6689802142394054760_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.5.126",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-16T18:14:24.167118355Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1705428851436210466
I0116 18:14:27.840655 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0116 18:14:27.840800 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0116 18:14:27.840905 140090947739648 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1808-1ajrg"
I0116 18:14:28.522598 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428867844-60f1419db8822-e0c9ee35-81da3665
I0116 18:14:41.582026 140090947739648 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1808-1ajrg"
I0116 18:14:42.016525 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428881585-60f141aad35f8-5881c5cf-48e687b2
I0116 18:14:44.364744 140090947739648 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1808-1ajrg"
I0116 18:14:45.012096 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428884370-60f141ad7b5d2-9bba8217-fe2c7e5d
I0116 18:14:47.287180 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0116 18:14:47.287425 140090947739648 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0116 18:14:47.287832 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:14:51.286008 140090947739648 xds_k8s_testcase.py:601] [psm-grpc-client-678cc7585b-8kks7] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-8d8f67d9d-h659f: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-8d8f67d9d-h659f: 100

I0116 18:14:51.286322 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 18:14:51.319887 140090947739648 grpc.py:75] [psm-grpc-client-678cc7585b-8kks7:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:14:55.326022 140090947739648 xds_k8s_testcase.py:601] [psm-grpc-client-678cc7585b-8kks7] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-8d8f67d9d-h659f: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-8d8f67d9d-h659f: 100

I0116 18:14:55.326307 140090947739648 xds_k8s_testcase.py:561] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1705428864117306229
I0116 18:14:55.326656 140090947739648 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0116 18:14:55.326818 140090947739648 xds_k8s_testcase.py:687] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0116 18:14:55.525845 140090947739648 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1808-1ajrg"
I0116 18:14:55.648733 140090947739648 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1808-1ajrg"
I0116 18:14:56.535553 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428895652-60f141b83d9a2-49f98dae-0b46ec71
I0116 18:15:09.577156 140090947739648 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1808-1ajrg"
I0116 18:15:09.971704 140090947739648 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1808-1ajrg"
I0116 18:15:10.316559 140090947739648 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1808-1ajrg"
I0116 18:15:10.987888 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428910319-60f141c63a76d-8c6d10d0-e2be72a9
I0116 18:15:13.357392 140090947739648 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1808-1ajrg"
I0116 18:15:13.777962 140090947739648 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1808-1ajrg"
I0116 18:15:14.357872 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428913783-60f141c9882df-bbd5492e-54254ac6
I0116 18:15:16.612009 140090947739648 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1808-1ajrg"
I0116 18:15:17.179127 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428916617-60f141cc3bfbb-1ddcc754-5e206dd9
I0116 18:15:49.525816 140090947739648 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1808-1ajrg"
I0116 18:15:50.226286 140090947739648 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1808-1ajrg"
I0116 18:15:50.781383 140090947739648 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1808-1ajrg"
I0116 18:15:51.312971 140090947739648 compute.py:614] Waiting 600 sec for compute operation id: operation-1705428950859-60f141ece3f5e-70016c20-cae89223
I0116 18:15:55.685087 140090947739648 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:16:05.862494 140090947739648 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:16:05.862853 140090947739648 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1808-1ajrg/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:16:06.507786 140090947739648 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:16:06.626513 140090947739648 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:16:06.626784 140090947739648 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1808-1ajrg
I0116 18:16:36.799577 140090947739648 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1808-1ajrg deleted
I0116 18:16:36.799879 140090947739648 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:16:46.983610 140090947739648 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:16:46.983917 140090947739648 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:16:47.094660 140090947739648 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:16:47.094928 140090947739648 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1808-1ajrg/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:16:47.629463 140090947739648 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:16:47.737637 140090947739648 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:16:47.737882 140090947739648 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1808-1ajrg
I0116 18:19:18.241648 140090947739648 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1808-1ajrg deleted
I0116 18:19:18.241998 140090947739648 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:19:18.242210 140090947739648 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-107cznmy:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1808-1ajrg%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-107cznmy%22;timeRange=2024-01-16T18:11:21.678656Z%2F2024-01-16T18:16:36.799846Z?project=grpc-testing
I0116 18:19:18.242332 140090947739648 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-2bzz5d6d:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1808-1ajrg%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-2bzz5d6d%22;timeRange=2024-01-16T18:10:15.022602Z%2F2024-01-16T18:19:18.241841Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
I0116 18:19:18.242536 140090947739648 base_testcase.py:52] ----- TestCase __main__.ApiListenerTest.test_api_listener PASSED -----
----------------------------------------------------------------------
Ran 1 test in 630.853s

OK
+ for test in "${test_suites[@]}"
+ run_test change_backend_service_test
+ local test_name=change_backend_service_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test'
+ set -x
+ python -m tests.change_backend_service_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:19:19.409697 140354170335232 xds_k8s_testcase.py:157] ----- Testing ChangeBackendServiceTest -----
I0116 18:19:19.409862 140354170335232 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:19:19.410054 140354170335232 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:19:19.424304 140354170335232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:19:19.439244 140354170335232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0116 18:19:19.443296 140354170335232 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1819-2mm1v
I0116 18:19:19.679557 140354170335232 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
I0116 18:19:19.892021 140354170335232 xds_k8s_testcase.py:663] Found unused xds port: 34951
I0116 18:19:19.894067 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0116 18:19:19.894216 140354170335232 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1819-2mm1v"
I0116 18:19:19.896402 140354170335232 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1819-2mm1v
type: GRPC
...

I0116 18:19:20.602124 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429159956-60f142b44d0d0-1002c6a0-7b58e996
I0116 18:19:22.984975 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0116 18:19:22.985218 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0116 18:19:22.985339 140354170335232 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1819-2mm1v"
I0116 18:19:22.991405 140354170335232 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1819-2mm1v
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1819-2mm1v
protocol: GRPC
...

I0116 18:19:23.781555 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429162994-60f142b732ba7-4ba3d9a4-6d495894
I0116 18:19:43.283302 140354170335232 traffic_director.py:309] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20240116-1819-2mm1v"
I0116 18:19:43.287459 140354170335232 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1819-2mm1v
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20240116-1819-2mm1v
protocol: GRPC
...

I0116 18:19:44.467094 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429183289-60f142ca8d98d-fdbb7b3a-c3a5652c
I0116 18:20:10.477612 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0116 18:20:10.477882 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0116 18:20:10.478011 140354170335232 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1819-2mm1v": psm-grpc-server:34951 -> psm-interop-backend-service-20240116-1819-2mm1v
I0116 18:20:10.486467 140354170335232 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1819-2mm1v
hostRules:
- hosts:
- psm-grpc-server:34951
pathMatcher: psm-interop-path-matcher-20240116-1819-2mm1v
name: psm-interop-url-map-20240116-1819-2mm1v
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1819-2mm1v
name: psm-interop-path-matcher-20240116-1819-2mm1v
...

I0116 18:20:11.222910 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429210488-60f142e47e01e-3fc6bc62-75be9ad9
I0116 18:20:13.517046 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0116 18:20:13.517320 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0116 18:20:13.517445 140354170335232 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1819-2mm1v proxy "GRPC" to URL map psm-interop-url-map-20240116-1819-2mm1v
I0116 18:20:13.519194 140354170335232 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1819-2mm1v
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1819-2mm1v
validate_for_proxyless: true
...

I0116 18:20:14.150188 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429213521-60f142e762598-f0ae0e30-62281096
I0116 18:20:16.505203 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0116 18:20:16.505461 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0116 18:20:16.505606 140354170335232 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1819-2mm1v" in network "default-vpc": 0.0.0.0:34951 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1819-2mm1v
I0116 18:20:16.507840 140354170335232 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1819-2mm1v
network: global/networks/default-vpc
portRange: 34951
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1819-2mm1v
...

I0116 18:20:17.241303 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429216509-60f142ea3bf1f-2481f0ad-246fa87f
I0116 18:20:30.239631 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0116 18:20:30.239901 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0116 18:20:30.240058 140354170335232 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1819-2mm1v: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 18:20:30.240310 140354170335232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1819-2mm1v%22;timeRange=2024-01-16T18:20:30.240165Z%2F2024-01-16T18:50:30.240165Z?project=grpc-testing
I0116 18:20:30.244499 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1819-2mm1v
labels:
name: psm-interop-server-20240116-1819-2mm1v
owner: xds-k8s-interop-test
...

I0116 18:20:30.379492 140354170335232 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1819-2mm1v created
I0116 18:20:30.382285 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1819-2mm1v-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:20:30.451729 140354170335232 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:20:30.451994 140354170335232 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1819-2mm1v
I0116 18:20:30.550683 140354170335232 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1819-2mm1v: neg_name=psm-interop-server-20240116-1819-2mm1v-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:20:30.550929 140354170335232 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1819-2mm1v/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:20:31.164003 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:20:31.220169 140354170335232 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:20:31.228308 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0hfmqqj0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0hfmqqj0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-0hfmqqj0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1819-2mm1v-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:20:31.299013 140354170335232 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:20:31.299281 140354170335232 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 18:20:31.405566 140354170335232 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-fcddd68b6-sflwf']
I0116 18:20:31.405838 140354170335232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-fcddd68b6-sflwf to start
I0116 18:20:34.646877 140354170335232 k8s_base_runner.py:820] Pod psm-grpc-server-fcddd68b6-sflwf ready, IP: 10.120.3.69
I0116 18:20:34.647170 140354170335232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-fcddd68b6-sflwf to change_backend_service_test/test_app_logs/psm-interop-server-20240116-1819-2mm1v_psm-grpc-server-fcddd68b6-sflwf.log
I0116 18:20:34.647840 140354077058624 k8s_log_collector.py:70] Starting log collection thread 140354077058624 for psm-grpc-server-fcddd68b6-sflwf
I0116 18:20:34.648372 140354170335232 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 18:20:40.989525 140354170335232 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 18:20:40.989990 140354170335232 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240116-1819-2mm1v: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 18:20:40.990217 140354170335232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1819-2mm1v%22;timeRange=2024-01-16T18:20:40.990118Z%2F2024-01-16T18:50:40.990118Z?project=grpc-testing
I0116 18:20:40.990303 140354170335232 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240116-1819-2mm1v
I0116 18:20:41.046823 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:20:41.091231 140354170335232 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0116 18:20:41.091476 140354170335232 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240116-1819-2mm1v
I0116 18:20:41.184950 140354170335232 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240116-1819-2mm1v: neg_name=psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0116 18:20:41.185177 140354170335232 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1819-2mm1v/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:20:41.828668 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:20:41.867642 140354170335232 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0116 18:20:41.875381 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1819-2mm1v
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-k3zxqu5r
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-k3zxqu5r
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-k3zxqu5r
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:20:41.925585 140354170335232 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0116 18:20:41.925842 140354170335232 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0116 18:20:42.072105 140354170335232 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-669d5c87d-4q7xc']
I0116 18:20:42.072440 140354170335232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-669d5c87d-4q7xc to start
I0116 18:20:45.306160 140354170335232 k8s_base_runner.py:820] Pod psm-grpc-server-alt-669d5c87d-4q7xc ready, IP: 10.120.0.29
I0116 18:20:45.306415 140354170335232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-669d5c87d-4q7xc to change_backend_service_test/test_app_logs/psm-interop-server-20240116-1819-2mm1v_psm-grpc-server-alt-669d5c87d-4q7xc.log
I0116 18:20:45.307032 140354068665920 k8s_log_collector.py:70] Starting log collection thread 140354068665920 for psm-grpc-server-alt-669d5c87d-4q7xc
I0116 18:20:45.307261 140354170335232 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0116 18:20:50.673354 140354170335232 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0116 18:20:50.673795 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0116 18:20:50.673914 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0116 18:20:50.710911 140354170335232 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:20:50.816988 140354170335232 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1819-2mm1v-psm-grpc-server" in zone us-central1-a
I0116 18:20:50.817241 140354170335232 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1819-2mm1v: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1819-2mm1v-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1819-2mm1v-psm-grpc-server', zone='us-central1-a')}
I0116 18:20:50.820651 140354170335232 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1819-2mm1v-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 18:20:51.780927 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429250822-60f1430af526d-e1dc62e2-22f0b9a2
I0116 18:21:28.105181 140354170335232 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1819-2mm1v to report all backends healthy: ['psm-interop-server-20240116-1819-2mm1v-psm-grpc-server']
I0116 18:22:02.058544 140354170335232 compute.py:494] Backend psm-interop-server-20240116-1819-2mm1v-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:22:06.145506 140354170335232 traffic_director.py:337] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:22:06.263615 140354170335232 traffic_director.py:340] Loaded NEG "psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt" in zone us-central1-a
I0116 18:22:06.263863 140354170335232 traffic_director.py:347] Adding backends to Backend Service psm-interop-backend-service-alt-20240116-1819-2mm1v: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt', zone='us-central1-a')}
I0116 18:22:06.267383 140354170335232 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1819-2mm1v-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0116 18:22:07.088771 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429326269-60f14352e8cd8-95fc2066-7f6890d5
I0116 18:23:03.189735 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0116 18:23:03.189914 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0116 18:23:03.189997 140354170335232 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1819-2mm1v: server_target=xds:///psm-grpc-server:34951 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 18:23:03.190189 140354170335232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1819-2mm1v%22;timeRange=2024-01-16T18:23:03.190051Z%2F2024-01-16T18:53:03.190051Z?project=grpc-testing
I0116 18:23:03.191823 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1819-2mm1v
labels:
name: psm-interop-client-20240116-1819-2mm1v
owner: xds-k8s-interop-test
...

I0116 18:23:03.252712 140354170335232 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1819-2mm1v created
I0116 18:23:03.252956 140354170335232 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1819-2mm1v/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:23:03.961810 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1819-2mm1v
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:23:04.022388 140354170335232 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:23:04.029098 140354170335232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1819-2mm1v
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-83z2dq66
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-83z2dq66
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-83z2dq66
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:34951"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:23:04.104819 140354170335232 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:23:04.105062 140354170335232 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:23:04.207811 140354170335232 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8676cbf886-4vxw5']
I0116 18:23:04.208084 140354170335232 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-8676cbf886-4vxw5 to start
I0116 18:23:07.438708 140354170335232 k8s_base_runner.py:820] Pod psm-grpc-client-8676cbf886-4vxw5 ready, IP: 10.120.7.33
I0116 18:23:07.438963 140354170335232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-8676cbf886-4vxw5 to change_backend_service_test/test_app_logs/psm-interop-client-20240116-1819-2mm1v_psm-grpc-client-8676cbf886-4vxw5.log
I0116 18:23:07.439515 140354060273216 k8s_log_collector.py:70] Starting log collection thread 140354060273216 for psm-grpc-client-8676cbf886-4vxw5
I0116 18:23:07.439791 140354170335232 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:23:08.580368 140354170335232 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:23:08.581044 140354170335232 client_app.py:284] [psm-grpc-client-8676cbf886-4vxw5] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:23:08.687769 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:23:18.732898 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:23:28.778634 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:23:38.824168 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:23:48.869437 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:24:04.919167 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:24:29.979639 140354170335232 client_app.py:312] [psm-grpc-client-8676cbf886-4vxw5] xDS control plane channel:
I0116 18:24:32.015143 140354170335232 client_app.py:323] [psm-grpc-client-8676cbf886-4vxw5] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:24:32.015388 140354170335232 client_app.py:294] [psm-grpc-client-8676cbf886-4vxw5] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:24:32.015682 140354170335232 client_app.py:244] [psm-grpc-client-8676cbf886-4vxw5] Waiting to report a READY channel to xds:///psm-grpc-server:34951
I0116 18:24:32.049077 140354170335232 client_app.py:371] [psm-grpc-client-8676cbf886-4vxw5] Server channel:
I0116 18:24:32.082658 140354170335232 client_app.py:384] [psm-grpc-client-8676cbf886-4vxw5] Found subchannel in state READY:
I0116 18:24:32.083125 140354170335232 client_app.py:255] [psm-grpc-client-8676cbf886-4vxw5] Channel to xds:///psm-grpc-server:34951 transitioned to state READY:
I0116 18:24:32.083303 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0116 18:24:32.083436 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0116 18:24:32.083643 140354170335232 grpc.py:75] [psm-grpc-client-8676cbf886-4vxw5:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:24:32.118381 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0116 18:24:32.118592 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0116 18:24:32.118792 140354170335232 grpc.py:75] [psm-grpc-client-8676cbf886-4vxw5:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:24:36.137404 140354170335232 xds_k8s_testcase.py:601] [psm-grpc-client-8676cbf886-4vxw5] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-fcddd68b6-sflwf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-fcddd68b6-sflwf: 100

I0116 18:24:36.137709 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0116 18:24:36.137835 140354170335232 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0116 18:24:36.137934 140354170335232 traffic_director.py:499] Patching URL map "psm-interop-url-map-20240116-1819-2mm1v": psm-grpc-server:34951 -> psm-interop-backend-service-alt-20240116-1819-2mm1v
I0116 18:24:36.141457 140354170335232 compute.py:544] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240116-1819-2mm1v
hostRules:
- hosts:
- psm-grpc-server:34951
pathMatcher: psm-interop-path-matcher-20240116-1819-2mm1v
name: psm-interop-url-map-20240116-1819-2mm1v
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240116-1819-2mm1v
name: psm-interop-path-matcher-20240116-1819-2mm1v
...

I0116 18:24:36.939114 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429476144-60f143e1d7635-431b2c84-f27dce79
I0116 18:25:02.895851 140354170335232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-669d5c87d-4q7xc']
I0116 18:25:02.896179 140354170335232 grpc.py:75] [psm-grpc-client-8676cbf886-4vxw5:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:25:06.896683 140354170335232 xds_k8s_testcase.py:601] [psm-grpc-client-8676cbf886-4vxw5] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-fcddd68b6-sflwf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-fcddd68b6-sflwf: 100

I0116 18:25:06.897259 140354170335232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-669d5c87d-4q7xc' not found in {'psm-grpc-server-fcddd68b6-sflwf': 100} : Server psm-grpc-server-alt-669d5c87d-4q7xc did not receive RPCs.
I0116 18:25:07.898564 140354170335232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-669d5c87d-4q7xc']
I0116 18:25:07.898972 140354170335232 grpc.py:75] [psm-grpc-client-8676cbf886-4vxw5:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:25:11.897559 140354170335232 xds_k8s_testcase.py:601] [psm-grpc-client-8676cbf886-4vxw5] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-fcddd68b6-sflwf: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-fcddd68b6-sflwf: 100

I0116 18:25:11.898015 140354170335232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-669d5c87d-4q7xc' not found in {'psm-grpc-server-fcddd68b6-sflwf': 100} : Server psm-grpc-server-alt-669d5c87d-4q7xc did not receive RPCs.
I0116 18:25:12.899318 140354170335232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-669d5c87d-4q7xc']
I0116 18:25:12.899683 140354170335232 grpc.py:75] [psm-grpc-client-8676cbf886-4vxw5:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:25:16.896406 140354170335232 xds_k8s_testcase.py:601] [psm-grpc-client-8676cbf886-4vxw5] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-669d5c87d-4q7xc: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-669d5c87d-4q7xc: 100

I0116 18:25:16.896910 140354170335232 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0116 18:25:16.897079 140354170335232 xds_k8s_testcase.py:687] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0116 18:25:17.119030 140354170335232 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1819-2mm1v"
I0116 18:25:17.921450 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429517122-60f14408ebc97-09e6a9cb-ca8cc743
I0116 18:25:30.864282 140354170335232 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1819-2mm1v"
I0116 18:25:30.960007 140354170335232 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1819-2mm1v"
I0116 18:25:31.368579 140354170335232 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1819-2mm1v"
I0116 18:25:31.805135 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429531371-60f1441682856-a35a0277-bf6bb86a
I0116 18:25:34.035328 140354170335232 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1819-2mm1v"
I0116 18:25:34.324041 140354170335232 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1819-2mm1v"
I0116 18:25:35.029857 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429534328-60f14419546f1-60fd96d6-4d7577fd
I0116 18:25:37.441382 140354170335232 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1819-2mm1v"
I0116 18:25:37.911149 140354170335232 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1819-2mm1v"
I0116 18:25:38.626510 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429537915-60f1441cc040c-db300d65-28f4dd9a
I0116 18:26:02.254816 140354170335232 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1819-2mm1v"
I0116 18:26:02.843783 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429562259-60f14433f780e-942bc454-6c4b8990
I0116 18:26:31.023869 140354170335232 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1819-2mm1v"
I0116 18:26:31.580890 140354170335232 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1819-2mm1v"
I0116 18:26:32.153951 140354170335232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429591692-60f14450093b3-c45d27fd-e34dfba2
I0116 18:26:36.569378 140354170335232 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:26:46.740069 140354170335232 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:26:46.740392 140354170335232 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1819-2mm1v/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:26:47.255329 140354170335232 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:26:47.382144 140354170335232 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:26:47.382418 140354170335232 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1819-2mm1v
I0116 18:27:17.598827 140354170335232 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1819-2mm1v deleted
I0116 18:27:17.599126 140354170335232 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:27:27.791649 140354170335232 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:27:27.791957 140354170335232 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:27:27.928151 140354170335232 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:27:27.928490 140354170335232 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1819-2mm1v/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:27:28.590055 140354170335232 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:27:28.711506 140354170335232 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:27:28.711786 140354170335232 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1819-2mm1v
I0116 18:29:59.233528 140354170335232 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1819-2mm1v deleted
I0116 18:29:59.233772 140354170335232 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0116 18:29:59.296749 140354170335232 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0116 18:29:59.358884 140354170335232 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1819-2mm1v/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:29:59.864275 140354170335232 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0116 18:29:59.928169 140354170335232 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1819-2mm1v
I0116 18:29:59.987109 140354170335232 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:29:59.987480 140354170335232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-83z2dq66:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1819-2mm1v%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-83z2dq66%22;timeRange=2024-01-16T18:23:03.190045Z%2F2024-01-16T18:27:17.599093Z?project=grpc-testing
I0116 18:29:59.987646 140354170335232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-0hfmqqj0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1819-2mm1v%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-0hfmqqj0%22;timeRange=2024-01-16T18:20:30.240155Z%2F2024-01-16T18:29:59.233741Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
I0116 18:29:59.987922 140354170335232 base_testcase.py:52] ----- TestCase __main__.ChangeBackendServiceTest.test_change_backend_service PASSED -----
----------------------------------------------------------------------
Ran 1 test in 640.581s

OK
+ for test in "${test_suites[@]}"
+ run_test custom_lb_test
+ local test_name=custom_lb_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test'
+ set -x
+ python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:30:01.293611 140680495173632 xds_k8s_testcase.py:157] ----- Testing CustomLbTest -----
I0116 18:30:01.293758 140680495173632 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:30:01.293930 140680495173632 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:30:01.308539 140680495173632 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:30:01.323758 140680495173632 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0116 18:30:01.328813 140680495173632 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1830-yh4hr
I0116 18:30:01.571133 140680495173632 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
I0116 18:30:01.940310 140680495173632 xds_k8s_testcase.py:663] Found unused xds port: 46866
I0116 18:30:01.940733 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0116 18:30:01.940829 140680495173632 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1830-yh4hr"
I0116 18:30:01.943378 140680495173632 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1830-yh4hr
type: GRPC
...

I0116 18:30:02.673666 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429802001-60f145189a59c-1c980be1-8edf4395
I0116 18:30:05.049734 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0116 18:30:05.050002 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0116 18:30:05.050151 140680495173632 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1830-yh4hr"
I0116 18:30:05.056786 140680495173632 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1830-yh4hr
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240116-1830-yh4hr
protocol: GRPC
...

I0116 18:30:05.851223 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429805058-60f1451b84b36-8506e6c3-3b1a40b4
I0116 18:30:27.548362 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0116 18:30:27.548651 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0116 18:30:27.548796 140680495173632 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1830-yh4hr": psm-grpc-server:46866 -> psm-interop-backend-service-20240116-1830-yh4hr
I0116 18:30:27.556795 140680495173632 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1830-yh4hr
hostRules:
- hosts:
- psm-grpc-server:46866
pathMatcher: psm-interop-path-matcher-20240116-1830-yh4hr
name: psm-interop-url-map-20240116-1830-yh4hr
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1830-yh4hr
name: psm-interop-path-matcher-20240116-1830-yh4hr
...

I0116 18:30:28.324588 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429827558-60f14530f9dba-50c44436-d887e143
I0116 18:30:30.567372 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0116 18:30:30.567605 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0116 18:30:30.567731 140680495173632 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1830-yh4hr proxy "GRPC" to URL map psm-interop-url-map-20240116-1830-yh4hr
I0116 18:30:30.569640 140680495173632 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1830-yh4hr
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1830-yh4hr
validate_for_proxyless: true
...

I0116 18:30:31.107448 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429830571-60f14533d96e6-ccc352b8-fbbd411c
I0116 18:30:33.367352 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0116 18:30:33.367596 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0116 18:30:33.367722 140680495173632 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1830-yh4hr" in network "default-vpc": 0.0.0.0:46866 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1830-yh4hr
I0116 18:30:33.369841 140680495173632 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1830-yh4hr
network: global/networks/default-vpc
portRange: 46866
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1830-yh4hr
...

I0116 18:30:34.178272 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429833371-60f1453685090-be58c742-9c2fd48c
I0116 18:31:02.242193 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0116 18:31:02.242478 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0116 18:31:02.242625 140680495173632 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1830-yh4hr: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 18:31:02.242892 140680495173632 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1830-yh4hr%22;timeRange=2024-01-16T18:31:02.242730Z%2F2024-01-16T19:01:02.242730Z?project=grpc-testing
I0116 18:31:02.247058 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1830-yh4hr
labels:
name: psm-interop-server-20240116-1830-yh4hr
owner: xds-k8s-interop-test
...

I0116 18:31:02.387445 140680495173632 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1830-yh4hr created
I0116 18:31:02.389925 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1830-yh4hr
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1830-yh4hr-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:31:02.468369 140680495173632 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:31:02.468701 140680495173632 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1830-yh4hr
I0116 18:31:02.576790 140680495173632 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1830-yh4hr: neg_name=psm-interop-server-20240116-1830-yh4hr-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:31:02.577035 140680495173632 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1830-yh4hr/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:31:03.201416 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1830-yh4hr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:31:03.267177 140680495173632 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:31:03.274495 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1830-yh4hr
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qtb1t0py
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qtb1t0py
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qtb1t0py
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1830-yh4hr-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:31:03.350711 140680495173632 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:31:03.350999 140680495173632 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 18:31:03.475745 140680495173632 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-567d995fc9-r42m6']
I0116 18:31:03.476088 140680495173632 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-567d995fc9-r42m6 to start
I0116 18:31:06.747008 140680495173632 k8s_base_runner.py:820] Pod psm-grpc-server-567d995fc9-r42m6 ready, IP: 10.120.0.30
I0116 18:31:06.747295 140680495173632 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-567d995fc9-r42m6 to custom_lb_test/test_app_logs/psm-interop-server-20240116-1830-yh4hr_psm-grpc-server-567d995fc9-r42m6.log
I0116 18:31:06.747903 140680402298432 k8s_log_collector.py:70] Starting log collection thread 140680402298432 for psm-grpc-server-567d995fc9-r42m6
I0116 18:31:06.748124 140680495173632 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 18:31:13.148179 140680495173632 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 18:31:13.148667 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0116 18:31:13.148808 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0116 18:31:13.192487 140680495173632 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:31:13.471592 140680495173632 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1830-yh4hr-psm-grpc-server" in zone us-central1-a
I0116 18:31:13.471844 140680495173632 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1830-yh4hr: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1830-yh4hr-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1830-yh4hr-psm-grpc-server', zone='us-central1-a')}
I0116 18:31:13.475430 140680495173632 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1830-yh4hr-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 18:31:14.387091 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705429873477-60f1455cc480d-515ca6c0-85b6a5b4
I0116 18:32:10.487518 140680495173632 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1830-yh4hr to report all backends healthy: ['psm-interop-server-20240116-1830-yh4hr-psm-grpc-server']
I0116 18:32:19.019460 140680495173632 compute.py:494] Backend psm-interop-server-20240116-1830-yh4hr-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:32:23.024167 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0116 18:32:23.024508 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0116 18:32:23.024729 140680495173632 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1830-yh4hr: server_target=xds:///psm-grpc-server:46866 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 18:32:23.024966 140680495173632 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1830-yh4hr%22;timeRange=2024-01-16T18:32:23.024826Z%2F2024-01-16T19:02:23.024826Z?project=grpc-testing
I0116 18:32:23.026712 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1830-yh4hr
labels:
name: psm-interop-client-20240116-1830-yh4hr
owner: xds-k8s-interop-test
...

I0116 18:32:23.939341 140680495173632 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1830-yh4hr created
I0116 18:32:23.939524 140680495173632 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1830-yh4hr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:32:24.533862 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1830-yh4hr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:32:24.592526 140680495173632 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:32:24.598968 140680495173632 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1830-yh4hr
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-frlal1en
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-frlal1en
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-frlal1en
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:46866"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:32:24.735771 140680495173632 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:32:24.736016 140680495173632 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:32:24.844158 140680495173632 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-55b6cc7f49-ghn84']
I0116 18:32:24.844502 140680495173632 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-55b6cc7f49-ghn84 to start
I0116 18:32:28.072386 140680495173632 k8s_base_runner.py:820] Pod psm-grpc-client-55b6cc7f49-ghn84 ready, IP: 10.120.7.34
I0116 18:32:28.072803 140680495173632 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-55b6cc7f49-ghn84 to custom_lb_test/test_app_logs/psm-interop-client-20240116-1830-yh4hr_psm-grpc-client-55b6cc7f49-ghn84.log
I0116 18:32:28.073435 140680393905728 k8s_log_collector.py:70] Starting log collection thread 140680393905728 for psm-grpc-client-55b6cc7f49-ghn84
I0116 18:32:28.073717 140680495173632 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:32:28.246692 140680495173632 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:32:28.247288 140680495173632 client_app.py:284] [psm-grpc-client-55b6cc7f49-ghn84] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:32:28.351938 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:32:38.397459 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:32:48.442630 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:32:58.487527 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:33:08.532227 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:33:24.581592 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:33:49.642080 140680495173632 client_app.py:312] [psm-grpc-client-55b6cc7f49-ghn84] xDS control plane channel:
I0116 18:33:51.677394 140680495173632 client_app.py:323] [psm-grpc-client-55b6cc7f49-ghn84] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:33:51.677732 140680495173632 client_app.py:294] [psm-grpc-client-55b6cc7f49-ghn84] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:33:51.678050 140680495173632 client_app.py:244] [psm-grpc-client-55b6cc7f49-ghn84] Waiting to report a READY channel to xds:///psm-grpc-server:46866
I0116 18:33:51.710697 140680495173632 client_app.py:371] [psm-grpc-client-55b6cc7f49-ghn84] Server channel:
I0116 18:33:51.743417 140680495173632 client_app.py:384] [psm-grpc-client-55b6cc7f49-ghn84] Found subchannel in state READY:
I0116 18:33:51.743920 140680495173632 client_app.py:255] [psm-grpc-client-55b6cc7f49-ghn84] Channel to xds:///psm-grpc-server:46866 transitioned to state READY:
I0116 18:33:51.744104 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0116 18:33:51.744211 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0116 18:33:51.744411 140680495173632 grpc.py:75] [psm-grpc-client-55b6cc7f49-ghn84:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:33:51.779422 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0116 18:33:51.779715 140680495173632 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0116 18:33:51.779972 140680495173632 grpc.py:75] [psm-grpc-client-55b6cc7f49-ghn84:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0116 18:34:01.823994 140680495173632 grpc.py:75] [psm-grpc-client-55b6cc7f49-ghn84:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), timeout=600, wait_for_ready=True)
I0116 18:34:01.860171 140680495173632 xds_k8s_testcase.py:398] [psm-grpc-client-55b6cc7f49-ghn84] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 251
result:
(15, DATA_LOSS): 251

I0116 18:34:01.860847 140680495173632 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0116 18:34:01.860999 140680495173632 xds_k8s_testcase.py:687] ----- TestMethod __main__.CustomLbTest.test_custom_lb_config teardown -----
I0116 18:34:02.033529 140680495173632 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1830-yh4hr"
I0116 18:34:02.841459 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430042037-60f145fd84d84-82427302-fe58309d
I0116 18:34:31.029229 140680495173632 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1830-yh4hr"
I0116 18:34:31.145626 140680495173632 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1830-yh4hr"
I0116 18:34:31.604668 140680495173632 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1830-yh4hr"
I0116 18:34:32.137222 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430071608-60f14619b8641-09a90212-b260c1c9
I0116 18:34:34.463322 140680495173632 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1830-yh4hr"
I0116 18:34:34.849951 140680495173632 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1830-yh4hr"
I0116 18:34:35.902663 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430074855-60f1461cd115d-5e236586-bea58dcc
I0116 18:34:38.156368 140680495173632 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1830-yh4hr"
I0116 18:34:38.631863 140680495173632 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1830-yh4hr"
I0116 18:34:39.255589 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430078637-60f146206c59b-2635c9b9-a57b7441
I0116 18:35:07.340169 140680495173632 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1830-yh4hr"
I0116 18:35:07.775269 140680495173632 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1830-yh4hr"
I0116 18:35:08.206880 140680495173632 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1830-yh4hr"
I0116 18:35:09.051403 140680495173632 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430108300-60f1463cb633e-03e97fc3-012a93cb
I0116 18:35:13.515139 140680495173632 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:35:23.709774 140680495173632 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:35:23.710109 140680495173632 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1830-yh4hr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:35:24.361899 140680495173632 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:35:24.485684 140680495173632 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:35:24.485951 140680495173632 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1830-yh4hr
I0116 18:35:54.664112 140680495173632 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1830-yh4hr deleted
I0116 18:35:54.664424 140680495173632 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:36:04.908992 140680495173632 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:36:04.909280 140680495173632 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:36:05.015619 140680495173632 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:36:05.015899 140680495173632 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1830-yh4hr/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:36:05.568518 140680495173632 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:36:05.675562 140680495173632 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:36:05.675832 140680495173632 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1830-yh4hr
I0116 18:38:36.156412 140680495173632 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1830-yh4hr deleted
I0116 18:38:36.156890 140680495173632 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:38:36.157155 140680495173632 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-frlal1en:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1830-yh4hr%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-frlal1en%22;timeRange=2024-01-16T18:32:23.024818Z%2F2024-01-16T18:35:54.664385Z?project=grpc-testing
I0116 18:38:36.157321 140680495173632 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-qtb1t0py:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1830-yh4hr%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-qtb1t0py%22;timeRange=2024-01-16T18:31:02.242723Z%2F2024-01-16T18:38:36.156718Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0116 18:38:36.157572 140680495173632 base_testcase.py:52] ----- TestCase __main__.CustomLbTest.test_custom_lb_config PASSED -----
----------------------------------------------------------------------
Ran 1 test in 514.867s

OK
+ for test in "${test_suites[@]}"
+ run_test failover_test
+ local test_name=failover_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test'
+ set -x
+ python -m tests.failover_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:38:37.354912 139916187795456 xds_k8s_testcase.py:157] ----- Testing FailoverTest -----
I0116 18:38:37.355074 139916187795456 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:38:37.355287 139916187795456 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:38:37.369989 139916187795456 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:38:37.384491 139916187795456 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0116 18:38:37.388688 139916187795456 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1838-3zmt3
I0116 18:38:37.599072 139916187795456 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
I0116 18:38:37.883034 139916187795456 xds_k8s_testcase.py:663] Found unused xds port: 43658
I0116 18:38:37.885246 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0116 18:38:37.885387 139916187795456 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1838-3zmt3"
I0116 18:38:37.887631 139916187795456 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1838-3zmt3
type: GRPC
...

I0116 18:38:38.559066 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430317941-60f14704a444b-f09596c3-5132c421
I0116 18:38:41.008606 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0116 18:38:41.008797 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0116 18:38:41.008881 139916187795456 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1838-3zmt3"
I0116 18:38:41.014888 139916187795456 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1838-3zmt3
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1838-3zmt3
protocol: GRPC
...

I0116 18:38:41.643981 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430321017-60f1470793143-9891e7fd-f12a1dec
I0116 18:39:07.751341 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0116 18:39:07.751630 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0116 18:39:07.751783 139916187795456 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1838-3zmt3": psm-grpc-server:43658 -> psm-interop-backend-service-20240116-1838-3zmt3
I0116 18:39:07.760278 139916187795456 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1838-3zmt3
hostRules:
- hosts:
- psm-grpc-server:43658
pathMatcher: psm-interop-path-matcher-20240116-1838-3zmt3
name: psm-interop-url-map-20240116-1838-3zmt3
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1838-3zmt3
name: psm-interop-path-matcher-20240116-1838-3zmt3
...

I0116 18:39:08.386746 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430347762-60f1472114c70-22765fe2-23b46cdd
I0116 18:39:10.707357 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0116 18:39:10.707584 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0116 18:39:10.707684 139916187795456 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1838-3zmt3 proxy "GRPC" to URL map psm-interop-url-map-20240116-1838-3zmt3
I0116 18:39:10.709325 139916187795456 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1838-3zmt3
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1838-3zmt3
validate_for_proxyless: true
...

I0116 18:39:11.311171 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430350711-60f14723e4a69-b97b15e5-9c3a1950
I0116 18:39:13.593915 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0116 18:39:13.594171 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0116 18:39:13.594320 139916187795456 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1838-3zmt3" in network "default-vpc": 0.0.0.0:43658 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1838-3zmt3
I0116 18:39:13.596307 139916187795456 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1838-3zmt3
network: global/networks/default-vpc
portRange: 43658
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1838-3zmt3
...

I0116 18:39:14.411430 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430353598-60f14726a56e6-c5318369-dacf2ea6
I0116 18:39:27.466271 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0116 18:39:27.466508 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0116 18:39:27.466614 139916187795456 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1838-3zmt3: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0116 18:39:27.466863 139916187795456 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1838-3zmt3%22;timeRange=2024-01-16T18:39:27.466723Z%2F2024-01-16T19:09:27.466723Z?project=grpc-testing
I0116 18:39:27.471076 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1838-3zmt3
labels:
name: psm-interop-server-20240116-1838-3zmt3
owner: xds-k8s-interop-test
...

I0116 18:39:27.607094 139916187795456 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1838-3zmt3 created
I0116 18:39:27.609537 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1838-3zmt3-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:39:27.667275 139916187795456 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:39:27.667496 139916187795456 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1838-3zmt3
I0116 18:39:27.759146 139916187795456 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1838-3zmt3: neg_name=psm-interop-server-20240116-1838-3zmt3-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:39:27.759376 139916187795456 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1838-3zmt3/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:39:28.519452 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:39:28.578206 139916187795456 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:39:28.585347 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-h64dejb9
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-h64dejb9
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-h64dejb9
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1838-3zmt3-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:39:28.650605 139916187795456 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:39:28.650878 139916187795456 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0116 18:39:28.806882 139916187795456 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:39:28.807202 139916187795456 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-64d4fd79d-7d5xg to start
I0116 18:39:32.034002 139916187795456 k8s_base_runner.py:820] Pod psm-grpc-server-64d4fd79d-7d5xg ready, IP: 10.120.4.148
I0116 18:39:32.034270 139916187795456 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-64d4fd79d-7d5xg to failover_test/test_app_logs/psm-interop-server-20240116-1838-3zmt3_psm-grpc-server-64d4fd79d-7d5xg.log
I0116 18:39:32.034872 139916094203456 k8s_log_collector.py:70] Starting log collection thread 139916094203456 for psm-grpc-server-64d4fd79d-7d5xg
I0116 18:39:32.035205 139916187795456 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-64d4fd79d-lsm29 to start
I0116 18:39:32.280364 139916187795456 k8s_base_runner.py:820] Pod psm-grpc-server-64d4fd79d-lsm29 ready, IP: 10.120.5.129
I0116 18:39:32.280617 139916187795456 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-64d4fd79d-lsm29 to failover_test/test_app_logs/psm-interop-server-20240116-1838-3zmt3_psm-grpc-server-64d4fd79d-lsm29.log
I0116 18:39:32.281208 139916085810752 k8s_log_collector.py:70] Starting log collection thread 139916085810752 for psm-grpc-server-64d4fd79d-lsm29
I0116 18:39:32.281447 139916187795456 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-64d4fd79d-sghqj to start
I0116 18:39:32.449963 139916187795456 k8s_base_runner.py:820] Pod psm-grpc-server-64d4fd79d-sghqj ready, IP: 10.120.2.150
I0116 18:39:32.450240 139916187795456 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-64d4fd79d-sghqj to failover_test/test_app_logs/psm-interop-server-20240116-1838-3zmt3_psm-grpc-server-64d4fd79d-sghqj.log
I0116 18:39:32.450812 139916008748608 k8s_log_collector.py:70] Starting log collection thread 139916008748608 for psm-grpc-server-64d4fd79d-sghqj
I0116 18:39:32.451083 139916187795456 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0116 18:39:38.890058 139916187795456 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0116 18:39:38.890652 139916187795456 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240116-1838-3zmt3: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 18:39:38.890875 139916187795456 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1838-3zmt3%22;timeRange=2024-01-16T18:39:38.890782Z%2F2024-01-16T19:09:38.890782Z?project=grpc-testing
I0116 18:39:38.892366 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1838-3zmt3
labels:
name: psm-interop-server-20240116-1838-3zmt3
owner: xds-k8s-interop-test
...

I0116 18:39:39.140897 139916187795456 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1838-3zmt3 created
I0116 18:39:39.143860 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:39:39.247310 139916187795456 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0116 18:39:39.247544 139916187795456 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240116-1838-3zmt3
I0116 18:39:39.407272 139916187795456 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240116-1838-3zmt3: neg_name=psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt, port=8080, zones=['us-west1-b']
I0116 18:39:39.407507 139916187795456 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1838-3zmt3/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:39:40.029449 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:39:40.121614 139916187795456 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0116 18:39:40.128218 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1838-3zmt3
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-42szsam1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-42szsam1
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-42szsam1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:39:40.229233 139916187795456 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0116 18:39:40.229459 139916187795456 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0116 18:39:40.404001 139916187795456 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:39:40.404234 139916187795456 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-78c95b8c9-n2ktt to start
I0116 18:39:45.959726 139916187795456 k8s_base_runner.py:820] Pod psm-grpc-server-alt-78c95b8c9-n2ktt ready, IP: 10.108.6.215
I0116 18:39:45.960041 139916187795456 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-78c95b8c9-n2ktt to failover_test/test_app_logs/psm-interop-server-20240116-1838-3zmt3_psm-grpc-server-alt-78c95b8c9-n2ktt.log
I0116 18:39:45.960716 139916000355904 k8s_log_collector.py:70] Starting log collection thread 139916000355904 for psm-grpc-server-alt-78c95b8c9-n2ktt
I0116 18:39:45.960992 139916187795456 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0116 18:39:51.627671 139916187795456 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0116 18:39:51.628118 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0116 18:39:51.628200 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0116 18:39:51.687085 139916187795456 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:39:51.796746 139916187795456 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1838-3zmt3-psm-grpc-server" in zone us-central1-a
I0116 18:39:51.796990 139916187795456 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1838-3zmt3: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1838-3zmt3-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server', zone='us-central1-a')}
I0116 18:39:51.800445 139916187795456 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server
maxRatePerEndpoint: 100
...

I0116 18:39:52.691603 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430391802-60f1474b14a4b-fbc05347-40768c6f
I0116 18:40:37.921702 139916187795456 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1838-3zmt3 to report all backends healthy: ['psm-interop-server-20240116-1838-3zmt3-psm-grpc-server']
I0116 18:41:08.230056 139916187795456 compute.py:494] Backend psm-interop-server-20240116-1838-3zmt3-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:41:12.305352 139916187795456 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:41:12.435618 139916187795456 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt" in zone us-west1-b
I0116 18:41:12.435904 139916187795456 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1838-3zmt3: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1838-3zmt3-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server', zone='us-central1-a')}
I0116 18:41:12.440018 139916187795456 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1838-3zmt3-psm-grpc-server
maxRatePerEndpoint: 100
...

I0116 18:41:13.610202 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430472442-60f14797fc120-1a4c5eec-51e0bc2e
I0116 18:41:58.974316 139916187795456 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1838-3zmt3 to report all backends healthy: ['psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt', 'psm-interop-server-20240116-1838-3zmt3-psm-grpc-server']
I0116 18:41:59.644878 139916187795456 compute.py:494] Backend psm-interop-server-20240116-1838-3zmt3-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:42:12.726310 139916187795456 compute.py:494] Backend psm-interop-server-20240116-1838-3zmt3-psm-grpc-server-alt in zone us-west1-b reported healthy
I0116 18:42:16.731032 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0116 18:42:16.731324 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0116 18:42:16.731470 139916187795456 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1838-3zmt3: server_target=xds:///psm-grpc-server:43658 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 18:42:16.731724 139916187795456 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1838-3zmt3%22;timeRange=2024-01-16T18:42:16.731563Z%2F2024-01-16T19:12:16.731563Z?project=grpc-testing
I0116 18:42:16.733462 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1838-3zmt3
labels:
name: psm-interop-client-20240116-1838-3zmt3
owner: xds-k8s-interop-test
...

I0116 18:42:16.892255 139916187795456 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1838-3zmt3 created
I0116 18:42:16.892550 139916187795456 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1838-3zmt3/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:42:17.492186 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1838-3zmt3
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:42:17.558933 139916187795456 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:42:17.565334 139916187795456 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1838-3zmt3
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-1xsd6qiu
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-1xsd6qiu
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-1xsd6qiu
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:43658"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:42:17.636789 139916187795456 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:42:17.637038 139916187795456 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:42:17.743492 139916187795456 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-68bb77f9b9-5xvcf']
I0116 18:42:17.743802 139916187795456 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-68bb77f9b9-5xvcf to start
I0116 18:42:20.972901 139916187795456 k8s_base_runner.py:820] Pod psm-grpc-client-68bb77f9b9-5xvcf ready, IP: 10.120.5.130
I0116 18:42:20.973174 139916187795456 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-68bb77f9b9-5xvcf to failover_test/test_app_logs/psm-interop-client-20240116-1838-3zmt3_psm-grpc-client-68bb77f9b9-5xvcf.log
I0116 18:42:20.973816 139915991963200 k8s_log_collector.py:70] Starting log collection thread 139915991963200 for psm-grpc-client-68bb77f9b9-5xvcf
I0116 18:42:20.974020 139916187795456 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:42:22.108534 139916187795456 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:42:22.109099 139916187795456 client_app.py:284] [psm-grpc-client-68bb77f9b9-5xvcf] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:42:22.213101 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:42:32.258490 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:42:42.303565 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:42:52.348643 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:43:02.393807 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:43:18.442918 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:43:43.502700 139916187795456 client_app.py:312] [psm-grpc-client-68bb77f9b9-5xvcf] xDS control plane channel:
I0116 18:43:45.537458 139916187795456 client_app.py:323] [psm-grpc-client-68bb77f9b9-5xvcf] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:43:45.537754 139916187795456 client_app.py:294] [psm-grpc-client-68bb77f9b9-5xvcf] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:43:45.537992 139916187795456 client_app.py:244] [psm-grpc-client-68bb77f9b9-5xvcf] Waiting to report a READY channel to xds:///psm-grpc-server:43658
I0116 18:43:45.570206 139916187795456 client_app.py:371] [psm-grpc-client-68bb77f9b9-5xvcf] Server channel:
I0116 18:43:45.602533 139916187795456 client_app.py:384] [psm-grpc-client-68bb77f9b9-5xvcf] Found subchannel in state READY:
I0116 18:43:45.603032 139916187795456 client_app.py:255] [psm-grpc-client-68bb77f9b9-5xvcf] Channel to xds:///psm-grpc-server:43658 transitioned to state READY:
I0116 18:43:45.603206 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0116 18:43:45.603301 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0116 18:43:45.603513 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 18:43:45.637841 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0116 18:43:45.638060 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0116 18:43:45.638365 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:43:45.638552 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:43:49.672961 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 34
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 34
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 33

I0116 18:43:49.673412 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0116 18:43:49.673561 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0116 18:43:49.673697 139916187795456 server_app.py:98] [psm-grpc-server-64d4fd79d-7d5xg] >> Setting health status to NOT_SERVING
I0116 18:43:49.674044 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-7d5xg:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:43:49.772350 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-7d5xg:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:43:49.805335 139916187795456 server_app.py:102] [psm-grpc-server-64d4fd79d-7d5xg] << Health status status: NOT_SERVING

I0116 18:43:49.805910 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:43:49.806124 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:43:53.832926 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 34

I0116 18:43:53.833475 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-7d5xg' not found in ['psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj'] : Unexpected server psm-grpc-server-64d4fd79d-7d5xg received RPCs.
I0116 18:43:54.834836 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:43:54.835226 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:43:58.832589 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 34
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 34
psm-grpc-server-64d4fd79d-sghqj: 33
psm-grpc-server-64d4fd79d-7d5xg: 33

I0116 18:43:58.833029 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-7d5xg' not found in ['psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj'] : Unexpected server psm-grpc-server-64d4fd79d-7d5xg received RPCs.
I0116 18:43:59.834354 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:43:59.834808 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:03.833770 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50

I0116 18:44:03.834144 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0116 18:44:03.834287 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0116 18:44:03.834369 139916187795456 server_app.py:98] [psm-grpc-server-64d4fd79d-lsm29] >> Setting health status to NOT_SERVING
I0116 18:44:03.834669 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-lsm29:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:44:03.933147 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-lsm29:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:44:03.966021 139916187795456 server_app.py:102] [psm-grpc-server-64d4fd79d-lsm29] << Health status status: NOT_SERVING

I0116 18:44:03.966550 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-sghqj', 'psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:03.966744 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:07.995310 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50

I0116 18:44:07.995881 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-78c95b8c9-n2ktt' not found in {'psm-grpc-server-64d4fd79d-lsm29': 50, 'psm-grpc-server-64d4fd79d-sghqj': 50} : Server psm-grpc-server-alt-78c95b8c9-n2ktt did not receive RPCs.
I0116 18:44:08.997195 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-sghqj', 'psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:08.997581 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:12.992619 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 50
psm-grpc-server-64d4fd79d-sghqj: 50

I0116 18:44:12.993036 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-78c95b8c9-n2ktt' not found in {'psm-grpc-server-64d4fd79d-lsm29': 50, 'psm-grpc-server-64d4fd79d-sghqj': 50} : Server psm-grpc-server-alt-78c95b8c9-n2ktt did not receive RPCs.
I0116 18:44:13.994342 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-sghqj', 'psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:13.994695 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:17.992300 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 20
psm-grpc-server-alt-78c95b8c9-n2ktt: 20
psm-grpc-server-64d4fd79d-sghqj: 60
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 20
psm-grpc-server-alt-78c95b8c9-n2ktt: 20
psm-grpc-server-64d4fd79d-sghqj: 60

I0116 18:44:17.992724 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-lsm29' not found in ['psm-grpc-server-64d4fd79d-sghqj', 'psm-grpc-server-alt-78c95b8c9-n2ktt'] : Unexpected server psm-grpc-server-64d4fd79d-lsm29 received RPCs.
I0116 18:44:18.994009 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-sghqj', 'psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:18.994365 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:22.992425 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 39
psm-grpc-server-64d4fd79d-sghqj: 61
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 39
psm-grpc-server-64d4fd79d-sghqj: 61

I0116 18:44:22.992869 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0116 18:44:22.992983 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0116 18:44:22.993040 139916187795456 server_app.py:98] [psm-grpc-server-64d4fd79d-sghqj] >> Setting health status to NOT_SERVING
I0116 18:44:22.993324 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-sghqj:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:44:23.091536 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-sghqj:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:44:23.124327 139916187795456 server_app.py:102] [psm-grpc-server-64d4fd79d-sghqj] << Health status status: NOT_SERVING

I0116 18:44:23.124983 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:23.125199 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:27.152951 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 34
psm-grpc-server-64d4fd79d-sghqj: 66
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 34
psm-grpc-server-64d4fd79d-sghqj: 66

I0116 18:44:27.153555 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-sghqj' not found in ['psm-grpc-server-alt-78c95b8c9-n2ktt'] : Unexpected server psm-grpc-server-64d4fd79d-sghqj received RPCs.
I0116 18:44:28.154869 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:28.155258 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:32.185096 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 72
psm-grpc-server-64d4fd79d-sghqj: 28
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 72
psm-grpc-server-64d4fd79d-sghqj: 28

I0116 18:44:32.185463 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-sghqj' not found in ['psm-grpc-server-alt-78c95b8c9-n2ktt'] : Unexpected server psm-grpc-server-64d4fd79d-sghqj received RPCs.
I0116 18:44:33.186747 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-78c95b8c9-n2ktt']
I0116 18:44:33.187149 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:37.224114 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 100

I0116 18:44:37.224562 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0116 18:44:37.224778 139916187795456 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0116 18:44:37.224879 139916187795456 server_app.py:89] [psm-grpc-server-64d4fd79d-7d5xg] >> Setting health status to SERVING
I0116 18:44:37.224995 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-7d5xg:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.257988 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-7d5xg:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.290531 139916187795456 server_app.py:91] [psm-grpc-server-64d4fd79d-7d5xg] << Health status status: SERVING

I0116 18:44:37.290842 139916187795456 server_app.py:89] [psm-grpc-server-64d4fd79d-lsm29] >> Setting health status to SERVING
I0116 18:44:37.291014 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-lsm29:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.324119 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-lsm29:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.357151 139916187795456 server_app.py:91] [psm-grpc-server-64d4fd79d-lsm29] << Health status status: SERVING

I0116 18:44:37.357421 139916187795456 server_app.py:89] [psm-grpc-server-64d4fd79d-sghqj] >> Setting health status to SERVING
I0116 18:44:37.357581 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-sghqj:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.392377 139916187795456 grpc.py:75] [psm-grpc-server-64d4fd79d-sghqj:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0116 18:44:37.425105 139916187795456 server_app.py:91] [psm-grpc-server-64d4fd79d-sghqj] << Health status status: SERVING

I0116 18:44:37.425615 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:44:37.425850 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:41.464961 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 100

I0116 18:44:41.465446 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-7d5xg' not found in {'psm-grpc-server-alt-78c95b8c9-n2ktt': 100} : Server psm-grpc-server-64d4fd79d-7d5xg did not receive RPCs.
I0116 18:44:42.466750 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:44:42.467138 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:46.473585 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-78c95b8c9-n2ktt: 84
psm-grpc-server-64d4fd79d-7d5xg: 16
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-78c95b8c9-n2ktt: 84
psm-grpc-server-64d4fd79d-7d5xg: 16

I0116 18:44:46.474117 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-64d4fd79d-lsm29' not found in {'psm-grpc-server-alt-78c95b8c9-n2ktt': 84, 'psm-grpc-server-64d4fd79d-7d5xg': 16} : Server psm-grpc-server-64d4fd79d-lsm29 did not receive RPCs.
I0116 18:44:47.475392 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:44:47.475805 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:51.473033 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-alt-78c95b8c9-n2ktt: 1
psm-grpc-server-64d4fd79d-sghqj: 32
psm-grpc-server-64d4fd79d-7d5xg: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-alt-78c95b8c9-n2ktt: 1
psm-grpc-server-64d4fd79d-sghqj: 32
psm-grpc-server-64d4fd79d-7d5xg: 34

I0116 18:44:51.473497 139916187795456 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-78c95b8c9-n2ktt' not found in ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj'] : Unexpected server psm-grpc-server-alt-78c95b8c9-n2ktt received RPCs.
I0116 18:44:52.474831 139916187795456 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-64d4fd79d-7d5xg', 'psm-grpc-server-64d4fd79d-lsm29', 'psm-grpc-server-64d4fd79d-sghqj']
I0116 18:44:52.475211 139916187795456 grpc.py:75] [psm-grpc-client-68bb77f9b9-5xvcf:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 18:44:56.472753 139916187795456 xds_k8s_testcase.py:601] [psm-grpc-client-68bb77f9b9-5xvcf] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-64d4fd79d-sghqj: 34
psm-grpc-server-64d4fd79d-7d5xg: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-64d4fd79d-lsm29: 33
psm-grpc-server-64d4fd79d-sghqj: 34
psm-grpc-server-64d4fd79d-7d5xg: 33

I0116 18:44:56.473160 139916187795456 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0116 18:44:56.473318 139916187795456 xds_k8s_testcase.py:687] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0116 18:44:56.718838 139916187795456 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1838-3zmt3"
I0116 18:44:57.549276 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430696722-60f1486de01cb-313d7030-597190af
I0116 18:45:06.344833 139916187795456 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1838-3zmt3"
I0116 18:45:06.454876 139916187795456 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1838-3zmt3"
I0116 18:45:06.869887 139916187795456 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1838-3zmt3"
I0116 18:45:07.437592 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430706872-60f148778e0ba-9d5c447e-c445298c
I0116 18:45:09.772494 139916187795456 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1838-3zmt3"
I0116 18:45:10.152072 139916187795456 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1838-3zmt3"
I0116 18:45:10.850225 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430710157-60f1487aafefe-74597a48-f17bef00
I0116 18:45:13.142231 139916187795456 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1838-3zmt3"
I0116 18:45:13.570308 139916187795456 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1838-3zmt3"
I0116 18:45:14.525472 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430713575-60f1487df289a-cba0bde1-ca1eb65e
I0116 18:45:38.280364 139916187795456 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1838-3zmt3"
I0116 18:45:38.715414 139916187795456 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1838-3zmt3"
I0116 18:45:39.089435 139916187795456 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1838-3zmt3"
I0116 18:45:39.576922 139916187795456 compute.py:614] Waiting 600 sec for compute operation id: operation-1705430739169-60f148965b163-016936eb-399da309
I0116 18:45:44.040132 139916187795456 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:45:54.237289 139916187795456 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:45:54.237646 139916187795456 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1838-3zmt3/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:45:54.729479 139916187795456 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:45:54.846813 139916187795456 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:45:54.847073 139916187795456 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1838-3zmt3
I0116 18:46:25.018340 139916187795456 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1838-3zmt3 deleted
I0116 18:46:25.018682 139916187795456 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:46:35.195657 139916187795456 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:46:35.195976 139916187795456 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:46:35.309354 139916187795456 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:46:35.309631 139916187795456 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1838-3zmt3/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:46:36.022166 139916187795456 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:46:36.137070 139916187795456 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:46:36.137331 139916187795456 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1838-3zmt3
I0116 18:49:06.655236 139916187795456 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1838-3zmt3 deleted
I0116 18:49:06.655524 139916187795456 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0116 18:49:17.072069 139916187795456 k8s_base_runner.py:723] Deployment psm-grpc-server-alt deleted
I0116 18:49:17.072406 139916187795456 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0116 18:49:17.267261 139916187795456 k8s_base_runner.py:739] Service psm-grpc-server-alt deleted
I0116 18:49:17.267531 139916187795456 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1838-3zmt3/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:49:17.859411 139916187795456 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0116 18:49:18.032901 139916187795456 k8s_base_runner.py:756] Service account psm-grpc-server-alt deleted
I0116 18:49:18.033152 139916187795456 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1838-3zmt3
I0116 18:51:18.640249 139916187795456 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1838-3zmt3 deleted
I0116 18:51:18.640803 139916187795456 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:51:18.641066 139916187795456 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-1xsd6qiu:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1838-3zmt3%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-1xsd6qiu%22;timeRange=2024-01-16T18:42:16.731556Z%2F2024-01-16T18:46:25.018646Z?project=grpc-testing
I0116 18:51:18.641237 139916187795456 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-h64dejb9:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1838-3zmt3%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-h64dejb9%22;timeRange=2024-01-16T18:39:27.466715Z%2F2024-01-16T18:49:06.655493Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
I0116 18:51:18.641504 139916187795456 base_testcase.py:52] ----- TestCase __main__.FailoverTest.test_failover PASSED -----
----------------------------------------------------------------------
Ran 1 test in 761.289s

OK
+ for test in "${test_suites[@]}"
+ run_test outlier_detection_test
+ local test_name=outlier_detection_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test'
+ set -x
+ python -m tests.outlier_detection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:51:19.767390 140136740319232 xds_k8s_testcase.py:157] ----- Testing OutlierDetectionTest -----
I0116 18:51:19.767590 140136740319232 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:51:19.767870 140136740319232 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:51:19.782107 140136740319232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:51:19.796745 140136740319232 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0116 18:51:19.800712 140136740319232 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1851-pxrm2
I0116 18:51:20.050420 140136740319232 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
I0116 18:51:20.388146 140136740319232 xds_k8s_testcase.py:663] Found unused xds port: 56853
I0116 18:51:20.388550 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0116 18:51:20.388662 140136740319232 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1851-pxrm2"
I0116 18:51:20.391337 140136740319232 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1851-pxrm2
type: GRPC
...

I0116 18:51:21.097071 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431080456-60f149dbd5160-9b50f294-0056f0cd
I0116 18:51:23.383810 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0116 18:51:23.384050 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0116 18:51:23.384164 140136740319232 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1851-pxrm2"
I0116 18:51:23.390313 140136740319232 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1851-pxrm2
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1851-pxrm2
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...

I0116 18:51:24.150934 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431083392-60f149dea1e14-9bfe2c6f-b3d3503c
I0116 18:52:03.073828 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0116 18:52:03.074110 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0116 18:52:03.074252 140136740319232 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1851-pxrm2": psm-grpc-server:56853 -> psm-interop-backend-service-20240116-1851-pxrm2
I0116 18:52:03.082563 140136740319232 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1851-pxrm2
hostRules:
- hosts:
- psm-grpc-server:56853
pathMatcher: psm-interop-path-matcher-20240116-1851-pxrm2
name: psm-interop-url-map-20240116-1851-pxrm2
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1851-pxrm2
name: psm-interop-path-matcher-20240116-1851-pxrm2
...

I0116 18:52:03.727043 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431123084-60f14a047c49a-f50d1027-68a48d53
I0116 18:52:06.065083 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0116 18:52:06.065314 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0116 18:52:06.065437 140136740319232 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1851-pxrm2 proxy "GRPC" to URL map psm-interop-url-map-20240116-1851-pxrm2
I0116 18:52:06.067231 140136740319232 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1851-pxrm2
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1851-pxrm2
validate_for_proxyless: true
...

I0116 18:52:06.481500 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431126069-60f14a0754fbe-5c4a1aaf-58eb87a5
I0116 18:52:08.775064 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0116 18:52:08.775389 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0116 18:52:08.775494 140136740319232 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1851-pxrm2" in network "default-vpc": 0.0.0.0:56853 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1851-pxrm2
I0116 18:52:08.777795 140136740319232 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1851-pxrm2
network: global/networks/default-vpc
portRange: 56853
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1851-pxrm2
...

I0116 18:52:09.558339 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431128779-60f14a09eab32-8504e96d-9cec5f14
I0116 18:52:56.990982 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0116 18:52:56.991243 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0116 18:52:56.991367 140136740319232 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1851-pxrm2: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0116 18:52:56.991581 140136740319232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1851-pxrm2%22;timeRange=2024-01-16T18:52:56.991450Z%2F2024-01-16T19:22:56.991450Z?project=grpc-testing
I0116 18:52:56.995350 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1851-pxrm2
labels:
name: psm-interop-server-20240116-1851-pxrm2
owner: xds-k8s-interop-test
...

I0116 18:52:57.150967 140136740319232 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1851-pxrm2 created
I0116 18:52:57.153510 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1851-pxrm2
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1851-pxrm2-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 18:52:57.216414 140136740319232 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 18:52:57.216689 140136740319232 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1851-pxrm2
I0116 18:52:57.306515 140136740319232 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1851-pxrm2: neg_name=psm-interop-server-20240116-1851-pxrm2-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 18:52:57.306776 140136740319232 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1851-pxrm2/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:52:57.977382 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1851-pxrm2
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:52:58.042477 140136740319232 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 18:52:58.049543 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1851-pxrm2
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snqok35v
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snqok35v
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snqok35v
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1851-pxrm2-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:52:58.113905 140136740319232 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 18:52:58.114128 140136740319232 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0116 18:52:58.280140 140136740319232 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:52:58.280573 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58d485cfc9-9mbvl to start
I0116 18:53:01.577372 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-server-58d485cfc9-9mbvl ready, IP: 10.120.2.152
I0116 18:53:01.577649 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58d485cfc9-9mbvl to outlier_detection_test/test_app_logs/psm-interop-server-20240116-1851-pxrm2_psm-grpc-server-58d485cfc9-9mbvl.log
I0116 18:53:01.578313 140136647484992 k8s_log_collector.py:70] Starting log collection thread 140136647484992 for psm-grpc-server-58d485cfc9-9mbvl
I0116 18:53:01.578697 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58d485cfc9-qtdpv to start
I0116 18:53:01.753292 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-server-58d485cfc9-qtdpv ready, IP: 10.120.0.33
I0116 18:53:01.753586 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58d485cfc9-qtdpv to outlier_detection_test/test_app_logs/psm-interop-server-20240116-1851-pxrm2_psm-grpc-server-58d485cfc9-qtdpv.log
I0116 18:53:01.754204 140136639092288 k8s_log_collector.py:70] Starting log collection thread 140136639092288 for psm-grpc-server-58d485cfc9-qtdpv
I0116 18:53:01.754584 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58d485cfc9-sdcxq to start
I0116 18:53:01.919324 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-server-58d485cfc9-sdcxq ready, IP: 10.120.4.150
I0116 18:53:01.919598 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58d485cfc9-sdcxq to outlier_detection_test/test_app_logs/psm-interop-server-20240116-1851-pxrm2_psm-grpc-server-58d485cfc9-sdcxq.log
I0116 18:53:01.920150 140136630699584 k8s_log_collector.py:70] Starting log collection thread 140136630699584 for psm-grpc-server-58d485cfc9-sdcxq
I0116 18:53:01.920392 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58d485cfc9-vfl2j to start
I0116 18:53:02.094416 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-server-58d485cfc9-vfl2j ready, IP: 10.120.7.36
I0116 18:53:02.094669 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58d485cfc9-vfl2j to outlier_detection_test/test_app_logs/psm-interop-server-20240116-1851-pxrm2_psm-grpc-server-58d485cfc9-vfl2j.log
I0116 18:53:02.095227 140136622306880 k8s_log_collector.py:70] Starting log collection thread 140136622306880 for psm-grpc-server-58d485cfc9-vfl2j
I0116 18:53:02.095486 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58d485cfc9-wd8s9 to start
I0116 18:53:02.193934 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-server-58d485cfc9-wd8s9 ready, IP: 10.120.5.132
I0116 18:53:02.194205 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58d485cfc9-wd8s9 to outlier_detection_test/test_app_logs/psm-interop-server-20240116-1851-pxrm2_psm-grpc-server-58d485cfc9-wd8s9.log
I0116 18:53:02.194851 140136613914176 k8s_log_collector.py:70] Starting log collection thread 140136613914176 for psm-grpc-server-58d485cfc9-wd8s9
I0116 18:53:02.195120 140136740319232 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0116 18:53:08.599442 140136740319232 k8s_base_runner.py:788] Deployment psm-grpc-server has 5 replicas available
I0116 18:53:08.600228 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0116 18:53:08.600317 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0116 18:53:08.637138 140136740319232 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 18:53:08.897300 140136740319232 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1851-pxrm2-psm-grpc-server" in zone us-central1-a
I0116 18:53:08.897547 140136740319232 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1851-pxrm2: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1851-pxrm2-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1851-pxrm2-psm-grpc-server', zone='us-central1-a')}
I0116 18:53:08.900949 140136740319232 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1851-pxrm2-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 18:53:09.956670 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431188902-60f14a43413c6-b440ecb3-71211650
I0116 18:53:44.311860 140136740319232 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1851-pxrm2 to report all backends healthy: ['psm-interop-server-20240116-1851-pxrm2-psm-grpc-server']
I0116 18:53:52.958326 140136740319232 compute.py:494] Backend psm-interop-server-20240116-1851-pxrm2-psm-grpc-server in zone us-central1-a reported healthy
I0116 18:53:56.963013 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0116 18:53:56.963322 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0116 18:53:56.963487 140136740319232 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1851-pxrm2: server_target=xds:///psm-grpc-server:56853 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0116 18:53:56.963791 140136740319232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1851-pxrm2%22;timeRange=2024-01-16T18:53:56.963592Z%2F2024-01-16T19:23:56.963592Z?project=grpc-testing
I0116 18:53:56.965590 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1851-pxrm2
labels:
name: psm-interop-client-20240116-1851-pxrm2
owner: xds-k8s-interop-test
...

I0116 18:53:57.007244 140136740319232 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1851-pxrm2 created
I0116 18:53:57.007511 140136740319232 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1851-pxrm2/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:53:57.548408 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1851-pxrm2
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 18:53:57.613953 140136740319232 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 18:53:57.620524 140136740319232 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1851-pxrm2
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wwlg2d4l
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wwlg2d4l
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wwlg2d4l
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:56853"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 18:53:57.685778 140136740319232 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 18:53:57.686028 140136740319232 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 18:53:57.786398 140136740319232 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-bf64c45c6-d42wx']
I0116 18:53:57.786643 140136740319232 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-bf64c45c6-d42wx to start
I0116 18:54:01.017350 140136740319232 k8s_base_runner.py:820] Pod psm-grpc-client-bf64c45c6-d42wx ready, IP: 10.120.3.73
I0116 18:54:01.017617 140136740319232 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-bf64c45c6-d42wx to outlier_detection_test/test_app_logs/psm-interop-client-20240116-1851-pxrm2_psm-grpc-client-bf64c45c6-d42wx.log
I0116 18:54:01.018243 140136604472896 k8s_log_collector.py:70] Starting log collection thread 140136604472896 for psm-grpc-client-bf64c45c6-d42wx
I0116 18:54:01.018630 140136740319232 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 18:54:02.236204 140136740319232 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 18:54:02.236820 140136740319232 client_app.py:284] [psm-grpc-client-bf64c45c6-d42wx] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 18:54:02.342231 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:54:12.387910 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:54:22.432991 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:54:32.478013 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:54:42.522978 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:54:58.572615 140136740319232 client_app.py:312] [psm-grpc-client-bf64c45c6-d42wx] xDS control plane channel:
I0116 18:55:00.608167 140136740319232 client_app.py:323] [psm-grpc-client-bf64c45c6-d42wx] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 18:55:00.608535 140136740319232 client_app.py:294] [psm-grpc-client-bf64c45c6-d42wx] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 18:55:00.608879 140136740319232 client_app.py:244] [psm-grpc-client-bf64c45c6-d42wx] Waiting to report a READY channel to xds:///psm-grpc-server:56853
I0116 18:55:00.641519 140136740319232 client_app.py:371] [psm-grpc-client-bf64c45c6-d42wx] Server channel:
I0116 18:55:00.674551 140136740319232 client_app.py:384] [psm-grpc-client-bf64c45c6-d42wx] Found subchannel in state READY:
I0116 18:55:00.675013 140136740319232 client_app.py:255] [psm-grpc-client-bf64c45c6-d42wx] Channel to xds:///psm-grpc-server:56853 transitioned to state READY:
I0116 18:55:00.675175 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0116 18:55:00.675270 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0116 18:55:00.675510 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0116 18:55:00.710165 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0116 18:55:00.710404 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0116 18:55:00.710693 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:00.710906 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:01.747838 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-wd8s9: 20
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-9mbvl: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-wd8s9: 20
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-9mbvl: 20

I0116 18:55:01.748311 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0116 18:55:01.748438 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0116 18:55:01.748715 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-58d485cfc9-9mbvl error-code-2'}]}), wait_for_ready=True, timeout=5)
I0116 18:55:01.781742 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:01.782039 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:02.817573 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 20
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-wd8s9: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-wd8s9: 20

I0116 18:55:02.818194 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 20 not less than or equal to 0 : Expected all RPCs to succeed: 20 of 100 failed.
I0116 18:55:03.819524 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:03.820050 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:04.846823 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:04.847173 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0116 18:55:04.847296 140136740319232 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0116 18:55:04.847453 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), wait_for_ready=True, timeout=5)
I0116 18:55:04.880325 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:04.880681 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:05.917441 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:05.918022 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:06.919332 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:06.919816 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:07.947614 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:07.948008 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:08.949309 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:08.949766 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:09.977431 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:09.977805 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:10.979127 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:10.979485 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:12.007622 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:12.007994 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:13.009278 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:13.009698 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:14.036493 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:14.036877 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:15.038189 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:15.038550 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:16.070472 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:16.070861 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:17.072212 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:17.072695 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:18.107248 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:18.107618 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:19.108924 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:19.109375 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:20.136389 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:20.136748 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:21.138003 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:21.138367 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:22.166032 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:22.166409 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:23.167738 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:23.168092 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:24.196775 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:24.197220 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:25.198511 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:25.198878 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:26.226742 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:26.227142 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:27.228489 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:27.228873 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:28.255641 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:28.256036 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:29.257342 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:29.257816 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:30.286840 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:30.287214 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:31.288554 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:31.288961 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:32.316326 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-vfl2j: 25
psm-grpc-server-58d485cfc9-qtdpv: 25
psm-grpc-server-58d485cfc9-sdcxq: 25
psm-grpc-server-58d485cfc9-wd8s9: 25

I0116 18:55:32.316772 140136740319232 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58d485cfc9-9mbvl' not found in {'psm-grpc-server-58d485cfc9-vfl2j': 25, 'psm-grpc-server-58d485cfc9-qtdpv': 25, 'psm-grpc-server-58d485cfc9-sdcxq': 25, 'psm-grpc-server-58d485cfc9-wd8s9': 25} : Server psm-grpc-server-58d485cfc9-9mbvl did not receive RPCs.
I0116 18:55:33.318153 140136740319232 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58d485cfc9-9mbvl', 'psm-grpc-server-58d485cfc9-qtdpv', 'psm-grpc-server-58d485cfc9-sdcxq', 'psm-grpc-server-58d485cfc9-vfl2j', 'psm-grpc-server-58d485cfc9-wd8s9']
I0116 18:55:33.318521 140136740319232 grpc.py:75] [psm-grpc-client-bf64c45c6-d42wx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 18:55:34.347652 140136740319232 xds_k8s_testcase.py:601] [psm-grpc-client-bf64c45c6-d42wx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58d485cfc9-9mbvl: 20
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-wd8s9: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-58d485cfc9-wd8s9: 20
psm-grpc-server-58d485cfc9-vfl2j: 20
psm-grpc-server-58d485cfc9-qtdpv: 20
psm-grpc-server-58d485cfc9-sdcxq: 20
psm-grpc-server-58d485cfc9-9mbvl: 20

I0116 18:55:34.348136 140136740319232 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0116 18:55:34.348263 140136740319232 xds_k8s_testcase.py:687] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0116 18:55:34.605654 140136740319232 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1851-pxrm2"
I0116 18:55:35.359115 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431334609-60f14ace3633d-d505fbf9-9602aa8a
I0116 18:55:48.389485 140136740319232 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1851-pxrm2"
I0116 18:55:48.480681 140136740319232 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1851-pxrm2"
I0116 18:55:48.901468 140136740319232 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1851-pxrm2"
I0116 18:55:49.424105 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431348904-60f14adbd82f8-e7bc2dd9-486106b3
I0116 18:55:51.694614 140136740319232 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1851-pxrm2"
I0116 18:55:52.057859 140136740319232 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1851-pxrm2"
I0116 18:55:52.722245 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431352062-60f14adedb28e-cf8d2d47-81c2f137
I0116 18:55:54.984695 140136740319232 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1851-pxrm2"
I0116 18:55:55.348618 140136740319232 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1851-pxrm2"
I0116 18:55:56.071918 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431355353-60f14ae1fea8d-6843aca8-97f4dcae
I0116 18:56:19.807182 140136740319232 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1851-pxrm2"
I0116 18:56:20.183949 140136740319232 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1851-pxrm2"
I0116 18:56:20.602202 140136740319232 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1851-pxrm2"
I0116 18:56:21.099857 140136740319232 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431380676-60f14afa25144-50d3e27c-dcc19756
I0116 18:56:25.577568 140136740319232 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 18:56:35.753659 140136740319232 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 18:56:35.754023 140136740319232 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1851-pxrm2/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:56:36.340270 140136740319232 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 18:56:36.469936 140136740319232 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 18:56:36.470132 140136740319232 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1851-pxrm2
I0116 18:57:06.630915 140136740319232 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1851-pxrm2 deleted
I0116 18:57:06.631203 140136740319232 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 18:57:16.897411 140136740319232 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 18:57:16.897752 140136740319232 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 18:57:17.032940 140136740319232 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 18:57:17.033213 140136740319232 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1851-pxrm2/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 18:57:17.668497 140136740319232 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 18:57:17.783466 140136740319232 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 18:57:17.783714 140136740319232 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1851-pxrm2
I0116 18:59:18.245370 140136740319232 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1851-pxrm2 deleted
I0116 18:59:18.245736 140136740319232 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 18:59:18.245953 140136740319232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-wwlg2d4l:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1851-pxrm2%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-wwlg2d4l%22;timeRange=2024-01-16T18:53:56.963583Z%2F2024-01-16T18:57:06.631165Z?project=grpc-testing
I0116 18:59:18.246068 140136740319232 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-snqok35v:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1851-pxrm2%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-snqok35v%22;timeRange=2024-01-16T18:52:56.991443Z%2F2024-01-16T18:59:18.245566Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
I0116 18:59:18.246263 140136740319232 base_testcase.py:52] ----- TestCase __main__.OutlierDetectionTest.test_outlier_detection PASSED -----
----------------------------------------------------------------------
Ran 1 test in 478.482s

OK
+ for test in "${test_suites[@]}"
+ run_test remove_neg_test
+ local test_name=remove_neg_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test'
+ set -x
+ python -m tests.remove_neg_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 18:59:19.373194 140410096758784 xds_k8s_testcase.py:157] ----- Testing RemoveNegTest -----
I0116 18:59:19.373415 140410096758784 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 18:59:19.373714 140410096758784 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 18:59:19.388639 140410096758784 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 18:59:19.402341 140410096758784 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0116 18:59:19.406136 140410096758784 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1859-t09bh
I0116 18:59:19.629419 140410096758784 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
I0116 18:59:19.980165 140410096758784 xds_k8s_testcase.py:663] Found unused xds port: 51916
I0116 18:59:19.982266 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0116 18:59:19.982414 140410096758784 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1859-t09bh"
I0116 18:59:19.984844 140410096758784 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1859-t09bh
type: GRPC
...

I0116 18:59:20.697712 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431560038-60f14ba532740-d7254786-f76609c9
I0116 18:59:23.080377 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0116 18:59:23.080599 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0116 18:59:23.080686 140410096758784 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1859-t09bh"
I0116 18:59:23.086702 140410096758784 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1859-t09bh
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1859-t09bh
protocol: GRPC
...

I0116 18:59:23.856800 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431563090-60f14ba81bad7-f2f2cb47-8041d252
I0116 18:59:39.089561 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0116 18:59:39.089840 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0116 18:59:39.089959 140410096758784 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1859-t09bh": psm-grpc-server:51916 -> psm-interop-backend-service-20240116-1859-t09bh
I0116 18:59:39.097698 140410096758784 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1859-t09bh
hostRules:
- hosts:
- psm-grpc-server:51916
pathMatcher: psm-interop-path-matcher-20240116-1859-t09bh
name: psm-interop-url-map-20240116-1859-t09bh
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1859-t09bh
name: psm-interop-path-matcher-20240116-1859-t09bh
...

I0116 18:59:39.710048 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431579100-60f14bb760391-4f8ca096-0105035c
I0116 18:59:42.067129 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0116 18:59:42.067352 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0116 18:59:42.067479 140410096758784 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1859-t09bh proxy "GRPC" to URL map psm-interop-url-map-20240116-1859-t09bh
I0116 18:59:42.069141 140410096758784 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1859-t09bh
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1859-t09bh
validate_for_proxyless: true
...

I0116 18:59:42.565103 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431582071-60f14bba35985-4ab5c56f-0fb0146b
I0116 18:59:44.839978 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0116 18:59:44.840267 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0116 18:59:44.840424 140410096758784 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1859-t09bh" in network "default-vpc": 0.0.0.0:51916 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1859-t09bh
I0116 18:59:44.842673 140410096758784 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1859-t09bh
network: global/networks/default-vpc
portRange: 51916
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1859-t09bh
...

I0116 18:59:45.621882 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431584844-60f14bbcdab7e-02e303b7-457ffc48
I0116 19:00:18.052255 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0116 19:00:18.052573 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0116 19:00:18.052674 140410096758784 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1859-t09bh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:00:18.052885 140410096758784 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1859-t09bh%22;timeRange=2024-01-16T19:00:18.052735Z%2F2024-01-16T19:30:18.052735Z?project=grpc-testing
I0116 19:00:18.056907 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1859-t09bh
labels:
name: psm-interop-server-20240116-1859-t09bh
owner: xds-k8s-interop-test
...

I0116 19:00:18.223148 140410096758784 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1859-t09bh created
I0116 19:00:18.225845 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1859-t09bh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1859-t09bh-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:00:18.303453 140410096758784 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:00:18.303694 140410096758784 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1859-t09bh
I0116 19:00:18.411750 140410096758784 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1859-t09bh: neg_name=psm-interop-server-20240116-1859-t09bh-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:00:18.411968 140410096758784 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1859-t09bh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:00:19.173305 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1859-t09bh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:00:19.239559 140410096758784 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:00:19.247032 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1859-t09bh
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hm7si8ic
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hm7si8ic
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hm7si8ic
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1859-t09bh-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:00:19.320649 140410096758784 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:00:19.320905 140410096758784 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:00:19.435009 140410096758784 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-58585748bf-49bqr']
I0116 19:00:19.435341 140410096758784 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-58585748bf-49bqr to start
I0116 19:00:22.704046 140410096758784 k8s_base_runner.py:820] Pod psm-grpc-server-58585748bf-49bqr ready, IP: 10.120.5.134
I0116 19:00:22.704313 140410096758784 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-58585748bf-49bqr to remove_neg_test/test_app_logs/psm-interop-server-20240116-1859-t09bh_psm-grpc-server-58585748bf-49bqr.log
I0116 19:00:22.705008 140410003908160 k8s_log_collector.py:70] Starting log collection thread 140410003908160 for psm-grpc-server-58585748bf-49bqr
I0116 19:00:22.705207 140410096758784 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:00:31.258572 140410096758784 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:00:31.258966 140410096758784 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240116-1859-t09bh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:00:31.259133 140410096758784 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1859-t09bh%22;timeRange=2024-01-16T19:00:31.259044Z%2F2024-01-16T19:30:31.259044Z?project=grpc-testing
I0116 19:00:31.259204 140410096758784 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240116-1859-t09bh
I0116 19:00:31.324800 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1859-t09bh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:00:31.378069 140410096758784 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0116 19:00:31.378315 140410096758784 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240116-1859-t09bh
I0116 19:00:31.487257 140410096758784 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240116-1859-t09bh: neg_name=psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0116 19:00:31.487494 140410096758784 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1859-t09bh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:00:32.113113 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1859-t09bh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:00:32.159848 140410096758784 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0116 19:00:32.166365 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240116-1859-t09bh
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ffppku97
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ffppku97
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-ffppku97
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:00:32.220024 140410096758784 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0116 19:00:32.220273 140410096758784 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0116 19:00:32.318153 140410096758784 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:00:32.318425 140410096758784 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-58b7cc75c7-qqdfv to start
I0116 19:00:35.652655 140410096758784 k8s_base_runner.py:820] Pod psm-grpc-server-alt-58b7cc75c7-qqdfv ready, IP: 10.120.4.152
I0116 19:00:35.652930 140410096758784 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-58b7cc75c7-qqdfv to remove_neg_test/test_app_logs/psm-interop-server-20240116-1859-t09bh_psm-grpc-server-alt-58b7cc75c7-qqdfv.log
I0116 19:00:35.653546 140409995515456 k8s_log_collector.py:70] Starting log collection thread 140409995515456 for psm-grpc-server-alt-58b7cc75c7-qqdfv
I0116 19:00:35.653966 140410096758784 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0116 19:00:43.159494 140410096758784 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0116 19:00:43.159910 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0116 19:00:43.160019 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0116 19:00:43.203635 140410096758784 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:00:43.315117 140410096758784 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1859-t09bh-psm-grpc-server" in zone us-central1-a
I0116 19:00:43.315380 140410096758784 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1859-t09bh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1859-t09bh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server', zone='us-central1-a')}
I0116 19:00:43.318961 140410096758784 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:00:44.444795 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431643320-60f14bf49f228-cfbd5f3c-e2aad9f5
I0116 19:01:31.753541 140410096758784 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1859-t09bh to report all backends healthy: ['psm-interop-server-20240116-1859-t09bh-psm-grpc-server']
I0116 19:01:40.487301 140410096758784 compute.py:494] Backend psm-interop-server-20240116-1859-t09bh-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:01:45.406709 140410096758784 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:01:45.688925 140410096758784 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt" in zone us-central1-a
I0116 19:01:45.689162 140410096758784 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1859-t09bh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1859-t09bh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt', zone='us-central1-a')}
I0116 19:01:45.692921 140410096758784 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0116 19:01:47.249176 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431705694-60f14c301b1fc-dc48d0a7-2e6c2e77
I0116 19:02:26.313248 140410096758784 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1859-t09bh to report all backends healthy: ['psm-interop-server-20240116-1859-t09bh-psm-grpc-server', 'psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt']
I0116 19:02:26.558762 140410096758784 compute.py:494] Backend psm-interop-server-20240116-1859-t09bh-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:02:30.729613 140410096758784 compute.py:494] Backend psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt in zone us-central1-a reported healthy
I0116 19:02:34.734319 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0116 19:02:34.734610 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0116 19:02:34.734840 140410096758784 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1859-t09bh: server_target=xds:///psm-grpc-server:51916 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:02:34.735099 140410096758784 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1859-t09bh%22;timeRange=2024-01-16T19:02:34.734941Z%2F2024-01-16T19:32:34.734941Z?project=grpc-testing
I0116 19:02:34.736965 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1859-t09bh
labels:
name: psm-interop-client-20240116-1859-t09bh
owner: xds-k8s-interop-test
...

I0116 19:02:34.804330 140410096758784 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1859-t09bh created
I0116 19:02:34.804649 140410096758784 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1859-t09bh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:02:35.413515 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1859-t09bh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:02:35.474343 140410096758784 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:02:35.480811 140410096758784 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1859-t09bh
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ox7f7yrz
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ox7f7yrz
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ox7f7yrz
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:51916"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:02:35.550811 140410096758784 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:02:35.551061 140410096758784 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:02:35.664551 140410096758784 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6bc59dcddd-kvmpt']
I0116 19:02:35.664872 140410096758784 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6bc59dcddd-kvmpt to start
I0116 19:02:38.889524 140410096758784 k8s_base_runner.py:820] Pod psm-grpc-client-6bc59dcddd-kvmpt ready, IP: 10.120.3.75
I0116 19:02:38.889808 140410096758784 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6bc59dcddd-kvmpt to remove_neg_test/test_app_logs/psm-interop-client-20240116-1859-t09bh_psm-grpc-client-6bc59dcddd-kvmpt.log
I0116 19:02:38.890431 140409987122752 k8s_log_collector.py:70] Starting log collection thread 140409987122752 for psm-grpc-client-6bc59dcddd-kvmpt
I0116 19:02:38.890677 140410096758784 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:02:40.031926 140410096758784 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:02:40.032508 140410096758784 client_app.py:284] [psm-grpc-client-6bc59dcddd-kvmpt] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 19:02:40.138541 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:02:50.184761 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:03:00.230505 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:03:10.276233 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:03:20.321537 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:03:36.371683 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:04:01.432822 140410096758784 client_app.py:312] [psm-grpc-client-6bc59dcddd-kvmpt] xDS control plane channel:
I0116 19:04:03.469154 140410096758784 client_app.py:323] [psm-grpc-client-6bc59dcddd-kvmpt] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 19:04:03.469473 140410096758784 client_app.py:294] [psm-grpc-client-6bc59dcddd-kvmpt] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 19:04:03.469750 140410096758784 client_app.py:244] [psm-grpc-client-6bc59dcddd-kvmpt] Waiting to report a READY channel to xds:///psm-grpc-server:51916
I0116 19:04:03.503098 140410096758784 client_app.py:371] [psm-grpc-client-6bc59dcddd-kvmpt] Server channel:
I0116 19:04:03.536580 140410096758784 client_app.py:384] [psm-grpc-client-6bc59dcddd-kvmpt] Found subchannel in state READY:
I0116 19:04:03.537066 140410096758784 client_app.py:255] [psm-grpc-client-6bc59dcddd-kvmpt] Channel to xds:///psm-grpc-server:51916 transitioned to state READY:
I0116 19:04:03.537232 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0116 19:04:03.537339 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0116 19:04:03.537542 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:04:03.572920 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0116 19:04:03.573142 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0116 19:04:03.573344 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:07.602875 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:04:07.603169 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0116 19:04:07.603286 140410096758784 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0116 19:04:07.603615 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr', 'psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:04:07.603768 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:11.603298 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:04:11.603900 140410096758784 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-58b7cc75c7-qqdfv' not found in {'psm-grpc-server-58585748bf-49bqr': 100} : Server psm-grpc-server-alt-58b7cc75c7-qqdfv did not receive RPCs.
I0116 19:04:12.605292 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr', 'psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:04:12.605660 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:16.602845 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:04:16.603226 140410096758784 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-58b7cc75c7-qqdfv' not found in {'psm-grpc-server-58585748bf-49bqr': 100} : Server psm-grpc-server-alt-58b7cc75c7-qqdfv did not receive RPCs.
I0116 19:04:17.604624 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr', 'psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:04:17.605113 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:21.602554 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:04:21.602953 140410096758784 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-58b7cc75c7-qqdfv' not found in {'psm-grpc-server-58585748bf-49bqr': 100} : Server psm-grpc-server-alt-58b7cc75c7-qqdfv did not receive RPCs.
I0116 19:04:22.604280 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr', 'psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:04:22.604768 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:26.603061 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:04:26.603427 140410096758784 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-58b7cc75c7-qqdfv' not found in {'psm-grpc-server-58585748bf-49bqr': 100} : Server psm-grpc-server-alt-58b7cc75c7-qqdfv did not receive RPCs.
I0116 19:04:27.604790 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr', 'psm-grpc-server-alt-58b7cc75c7-qqdfv']
I0116 19:04:27.605226 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:04:31.603170 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-58b7cc75c7-qqdfv: 51
psm-grpc-server-58585748bf-49bqr: 49
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-58b7cc75c7-qqdfv: 51
psm-grpc-server-58585748bf-49bqr: 49

I0116 19:04:31.746869 140410096758784 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:04:31.863059 140410096758784 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1859-t09bh-psm-grpc-server-alt" in zone us-central1-a
I0116 19:04:31.863313 140410096758784 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1859-t09bh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1859-t09bh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server', zone='us-central1-a')}
I0116 19:04:31.867031 140410096758784 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1859-t09bh-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:04:32.893507 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431871869-60f14cce950b6-db85c2df-3f632d39
I0116 19:05:09.400500 140410096758784 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-58585748bf-49bqr']
I0116 19:05:09.400913 140410096758784 grpc.py:75] [psm-grpc-client-6bc59dcddd-kvmpt:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:05:13.402832 140410096758784 xds_k8s_testcase.py:601] [psm-grpc-client-6bc59dcddd-kvmpt] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58585748bf-49bqr: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-58585748bf-49bqr: 100

I0116 19:05:13.403251 140410096758784 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0116 19:05:13.403400 140410096758784 xds_k8s_testcase.py:687] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0116 19:05:13.534439 140410096758784 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1859-t09bh"
I0116 19:05:14.155181 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431913538-60f14cf65223c-8c46c9a8-6d4e8452
I0116 19:05:40.177185 140410096758784 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1859-t09bh"
I0116 19:05:40.289785 140410096758784 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1859-t09bh"
I0116 19:05:40.689279 140410096758784 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1859-t09bh"
I0116 19:05:41.292857 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431940692-60f14d1037aae-e179bc78-ea001ec2
I0116 19:05:43.550386 140410096758784 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1859-t09bh"
I0116 19:05:43.846104 140410096758784 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1859-t09bh"
I0116 19:05:44.522786 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431943850-60f14d133ab10-63f8fe8f-20316d82
I0116 19:05:46.836040 140410096758784 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1859-t09bh"
I0116 19:05:47.290375 140410096758784 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1859-t09bh"
I0116 19:05:47.914706 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431947295-60f14d1683ab8-b4752ca0-6feddc1b
I0116 19:06:13.619550 140410096758784 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1859-t09bh"
I0116 19:06:14.088524 140410096758784 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1859-t09bh"
I0116 19:06:14.468747 140410096758784 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1859-t09bh"
I0116 19:06:14.953971 140410096758784 compute.py:614] Waiting 600 sec for compute operation id: operation-1705431974547-60f14d3080fb6-5ddb0469-00e486f8
I0116 19:06:19.350299 140410096758784 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:06:29.525244 140410096758784 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:06:29.525557 140410096758784 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1859-t09bh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:06:30.128948 140410096758784 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:06:30.253390 140410096758784 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:06:30.253666 140410096758784 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1859-t09bh
I0116 19:07:00.484911 140410096758784 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1859-t09bh deleted
I0116 19:07:00.485203 140410096758784 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:07:10.649703 140410096758784 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:07:10.650017 140410096758784 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:07:10.764920 140410096758784 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:07:10.765198 140410096758784 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1859-t09bh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:07:11.361266 140410096758784 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:07:11.475247 140410096758784 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:07:11.475497 140410096758784 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1859-t09bh
I0116 19:08:11.721328 140410096758784 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1859-t09bh deleted
I0116 19:08:11.721543 140410096758784 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0116 19:08:11.777422 140410096758784 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0116 19:08:11.832836 140410096758784 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1859-t09bh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:08:12.415666 140410096758784 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0116 19:08:12.468746 140410096758784 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1859-t09bh
I0116 19:08:12.504347 140410096758784 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 19:08:12.504788 140410096758784 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-ox7f7yrz:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1859-t09bh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-ox7f7yrz%22;timeRange=2024-01-16T19:02:34.734933Z%2F2024-01-16T19:07:00.485168Z?project=grpc-testing
I0116 19:08:12.504959 140410096758784 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-hm7si8ic:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1859-t09bh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-hm7si8ic%22;timeRange=2024-01-16T19:00:18.052728Z%2F2024-01-16T19:08:11.721515Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
I0116 19:08:12.505198 140410096758784 base_testcase.py:52] ----- TestCase __main__.RemoveNegTest.test_remove_neg PASSED -----
----------------------------------------------------------------------
Ran 1 test in 533.135s

OK
+ for test in "${test_suites[@]}"
+ run_test round_robin_test
+ local test_name=round_robin_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test'
+ set -x
+ python -m tests.round_robin_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 19:08:13.655424 140476350898176 xds_k8s_testcase.py:157] ----- Testing RoundRobinTest -----
I0116 19:08:13.655550 140476350898176 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 19:08:13.655723 140476350898176 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 19:08:13.670344 140476350898176 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 19:08:13.684971 140476350898176 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0116 19:08:13.688730 140476350898176 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240116-1908-8kgir
I0116 19:08:13.909553 140476350898176 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
I0116 19:08:14.227509 140476350898176 xds_k8s_testcase.py:663] Found unused xds port: 32299
I0116 19:08:14.227943 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0116 19:08:14.228087 140476350898176 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1908-8kgir"
I0116 19:08:14.230617 140476350898176 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240116-1908-8kgir
type: GRPC
...

I0116 19:08:14.962813 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432094307-60f14da2b74d0-98fd4f95-d6d0678e
I0116 19:08:17.348668 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0116 19:08:17.348949 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0116 19:08:17.349082 140476350898176 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1908-8kgir"
I0116 19:08:17.355065 140476350898176 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1908-8kgir
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1908-8kgir
protocol: GRPC
...

I0116 19:08:18.059385 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432097357-60f14da59fe6c-7fff8443-80daa752
I0116 19:08:37.449301 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0116 19:08:37.449559 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0116 19:08:37.449681 140476350898176 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1908-8kgir": psm-grpc-server:32299 -> psm-interop-backend-service-20240116-1908-8kgir
I0116 19:08:37.457753 140476350898176 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1908-8kgir
hostRules:
- hosts:
- psm-grpc-server:32299
pathMatcher: psm-interop-path-matcher-20240116-1908-8kgir
name: psm-interop-url-map-20240116-1908-8kgir
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1908-8kgir
name: psm-interop-path-matcher-20240116-1908-8kgir
...

I0116 19:08:38.161715 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432117460-60f14db8cbe29-69133c6f-66a82f86
I0116 19:08:40.400409 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0116 19:08:40.400690 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0116 19:08:40.400803 140476350898176 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1908-8kgir proxy "GRPC" to URL map psm-interop-url-map-20240116-1908-8kgir
I0116 19:08:40.402456 140476350898176 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1908-8kgir
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1908-8kgir
validate_for_proxyless: true
...

I0116 19:08:41.001793 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432120404-60f14dbb9a973-4f8e6b1a-ec752fa8
I0116 19:08:43.263014 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0116 19:08:43.263249 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0116 19:08:43.263362 140476350898176 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1908-8kgir" in network "default-vpc": 0.0.0.0:32299 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1908-8kgir
I0116 19:08:43.265385 140476350898176 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1908-8kgir
network: global/networks/default-vpc
portRange: 32299
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1908-8kgir
...

I0116 19:08:43.996004 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432123267-60f14dbe5593c-4304ccf9-9e256003
I0116 19:08:57.036570 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0116 19:08:57.036821 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0116 19:08:57.036936 140476350898176 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1908-8kgir: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0116 19:08:57.037172 140476350898176 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1908-8kgir%22;timeRange=2024-01-16T19:08:57.037038Z%2F2024-01-16T19:38:57.037038Z?project=grpc-testing
I0116 19:08:57.040883 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1908-8kgir
labels:
name: psm-interop-server-20240116-1908-8kgir
owner: xds-k8s-interop-test
...

I0116 19:08:57.210040 140476350898176 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1908-8kgir created
I0116 19:08:57.212404 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1908-8kgir
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1908-8kgir-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:08:57.275961 140476350898176 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:08:57.276202 140476350898176 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1908-8kgir
I0116 19:08:57.375612 140476350898176 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1908-8kgir: neg_name=psm-interop-server-20240116-1908-8kgir-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:08:57.375855 140476350898176 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1908-8kgir/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:08:57.987459 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1908-8kgir
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:08:58.051575 140476350898176 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:08:58.058479 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1908-8kgir
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bjsyu1nt
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bjsyu1nt
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bjsyu1nt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1908-8kgir-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:08:58.130300 140476350898176 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:08:58.130541 140476350898176 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0116 19:08:58.267701 140476350898176 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-6d8c66f8b6-f6kqq', 'psm-grpc-server-6d8c66f8b6-w4rjr']
I0116 19:08:58.268057 140476350898176 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6d8c66f8b6-f6kqq to start
I0116 19:09:01.496417 140476350898176 k8s_base_runner.py:820] Pod psm-grpc-server-6d8c66f8b6-f6kqq ready, IP: 10.120.3.76
I0116 19:09:01.496688 140476350898176 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6d8c66f8b6-f6kqq to round_robin_test/test_app_logs/psm-interop-server-20240116-1908-8kgir_psm-grpc-server-6d8c66f8b6-f6kqq.log
I0116 19:09:01.497324 140476257134144 k8s_log_collector.py:70] Starting log collection thread 140476257134144 for psm-grpc-server-6d8c66f8b6-f6kqq
I0116 19:09:01.497528 140476350898176 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6d8c66f8b6-w4rjr to start
I0116 19:09:01.665822 140476350898176 k8s_base_runner.py:820] Pod psm-grpc-server-6d8c66f8b6-w4rjr ready, IP: 10.120.4.153
I0116 19:09:01.666101 140476350898176 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6d8c66f8b6-w4rjr to round_robin_test/test_app_logs/psm-interop-server-20240116-1908-8kgir_psm-grpc-server-6d8c66f8b6-w4rjr.log
I0116 19:09:01.666586 140476248741440 k8s_log_collector.py:70] Starting log collection thread 140476248741440 for psm-grpc-server-6d8c66f8b6-w4rjr
I0116 19:09:01.666834 140476350898176 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0116 19:09:09.162039 140476350898176 k8s_base_runner.py:788] Deployment psm-grpc-server has 2 replicas available
I0116 19:09:09.162526 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0116 19:09:09.162672 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0116 19:09:09.210783 140476350898176 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:09:09.329997 140476350898176 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1908-8kgir-psm-grpc-server" in zone us-central1-a
I0116 19:09:09.330274 140476350898176 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1908-8kgir: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1908-8kgir-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1908-8kgir-psm-grpc-server', zone='us-central1-a')}
I0116 19:09:09.333807 140476350898176 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1908-8kgir-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:09:10.377192 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432149335-60f14dd731f31-16f394bd-3eb17fe9
I0116 19:09:55.607328 140476350898176 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1908-8kgir to report all backends healthy: ['psm-interop-server-20240116-1908-8kgir-psm-grpc-server']
I0116 19:10:04.131375 140476350898176 compute.py:494] Backend psm-interop-server-20240116-1908-8kgir-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:10:08.136085 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0116 19:10:08.136404 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0116 19:10:08.136598 140476350898176 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1908-8kgir: server_target=xds:///psm-grpc-server:32299 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:10:08.136824 140476350898176 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1908-8kgir%22;timeRange=2024-01-16T19:10:08.136678Z%2F2024-01-16T19:40:08.136678Z?project=grpc-testing
I0116 19:10:08.138953 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1908-8kgir
labels:
name: psm-interop-client-20240116-1908-8kgir
owner: xds-k8s-interop-test
...

I0116 19:10:08.203889 140476350898176 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1908-8kgir created
I0116 19:10:08.204105 140476350898176 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1908-8kgir/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:10:08.788056 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1908-8kgir
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:10:08.845290 140476350898176 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:10:08.852025 140476350898176 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1908-8kgir
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-05hlokk9
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-05hlokk9
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-05hlokk9
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:32299"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:10:08.923767 140476350898176 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:10:08.924041 140476350898176 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:10:09.030906 140476350898176 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-774d8645f8-5459t']
I0116 19:10:09.031203 140476350898176 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-774d8645f8-5459t to start
I0116 19:10:12.266792 140476350898176 k8s_base_runner.py:820] Pod psm-grpc-client-774d8645f8-5459t ready, IP: 10.120.7.38
I0116 19:10:12.267045 140476350898176 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-774d8645f8-5459t to round_robin_test/test_app_logs/psm-interop-client-20240116-1908-8kgir_psm-grpc-client-774d8645f8-5459t.log
I0116 19:10:12.267424 140476166436416 k8s_log_collector.py:70] Starting log collection thread 140476166436416 for psm-grpc-client-774d8645f8-5459t
I0116 19:10:12.267508 140476350898176 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:10:13.410335 140476350898176 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:10:13.411107 140476350898176 client_app.py:284] [psm-grpc-client-774d8645f8-5459t] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0116 19:10:13.517838 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:10:23.562736 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:10:33.607440 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:10:43.652537 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:10:53.697018 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:11:09.746425 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:11:34.805982 140476350898176 client_app.py:312] [psm-grpc-client-774d8645f8-5459t] xDS control plane channel:
I0116 19:11:36.841046 140476350898176 client_app.py:323] [psm-grpc-client-774d8645f8-5459t] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0116 19:11:36.841378 140476350898176 client_app.py:294] [psm-grpc-client-774d8645f8-5459t] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0116 19:11:36.841650 140476350898176 client_app.py:244] [psm-grpc-client-774d8645f8-5459t] Waiting to report a READY channel to xds:///psm-grpc-server:32299
I0116 19:11:36.874076 140476350898176 client_app.py:371] [psm-grpc-client-774d8645f8-5459t] Server channel:
I0116 19:11:36.906331 140476350898176 client_app.py:384] [psm-grpc-client-774d8645f8-5459t] Found subchannel in state READY:
I0116 19:11:36.906800 140476350898176 client_app.py:255] [psm-grpc-client-774d8645f8-5459t] Channel to xds:///psm-grpc-server:32299 transitioned to state READY:
I0116 19:11:36.906950 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0116 19:11:36.907053 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0116 19:11:36.907261 140476350898176 grpc.py:75] [psm-grpc-client-774d8645f8-5459t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:11:36.941523 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0116 19:11:36.941780 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0116 19:11:36.942006 140476350898176 grpc.py:75] [psm-grpc-client-774d8645f8-5459t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:11:40.972398 140476350898176 xds_k8s_testcase.py:601] [psm-grpc-client-774d8645f8-5459t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d8c66f8b6-f6kqq: 50
psm-grpc-server-6d8c66f8b6-w4rjr: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d8c66f8b6-f6kqq: 50
psm-grpc-server-6d8c66f8b6-w4rjr: 50

I0116 19:11:40.972741 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0116 19:11:40.972909 140476350898176 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0116 19:11:40.973054 140476350898176 grpc.py:75] [psm-grpc-client-774d8645f8-5459t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0116 19:11:44.972228 140476350898176 xds_k8s_testcase.py:601] [psm-grpc-client-774d8645f8-5459t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d8c66f8b6-f6kqq: 50
psm-grpc-server-6d8c66f8b6-w4rjr: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d8c66f8b6-f6kqq: 50
psm-grpc-server-6d8c66f8b6-w4rjr: 50

I0116 19:11:44.972554 140476350898176 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0116 19:11:44.972709 140476350898176 xds_k8s_testcase.py:687] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0116 19:11:45.187460 140476350898176 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1908-8kgir"
I0116 19:11:45.875159 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432305191-60f14e6bd48bf-0df3bf7b-46c4954c
I0116 19:11:59.034265 140476350898176 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1908-8kgir"
I0116 19:11:59.140848 140476350898176 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1908-8kgir"
I0116 19:11:59.617877 140476350898176 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1908-8kgir"
I0116 19:12:00.074859 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432319621-60f14e799785b-75e8e6db-1b8e5eee
I0116 19:12:02.335093 140476350898176 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1908-8kgir"
I0116 19:12:02.547057 140476350898176 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1908-8kgir"
I0116 19:12:03.198424 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432322552-60f14e7c6322c-cc6f21fc-f584c40e
I0116 19:12:05.576387 140476350898176 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1908-8kgir"
I0116 19:12:06.006327 140476350898176 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1908-8kgir"
I0116 19:12:06.741236 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432326011-60f14e7faf8e7-492bbadc-7443ad3e
I0116 19:12:30.485371 140476350898176 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1908-8kgir"
I0116 19:12:30.882298 140476350898176 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1908-8kgir"
I0116 19:12:31.268491 140476350898176 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1908-8kgir"
I0116 19:12:31.766575 140476350898176 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432351342-60f14e97d7ec5-4f4282a0-6c8cdd12
I0116 19:12:36.194198 140476350898176 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:12:46.366366 140476350898176 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:12:46.366711 140476350898176 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1908-8kgir/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:12:46.987114 140476350898176 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:12:47.106771 140476350898176 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:12:47.107019 140476350898176 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1908-8kgir
I0116 19:13:17.276672 140476350898176 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1908-8kgir deleted
I0116 19:13:17.276885 140476350898176 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:13:27.453572 140476350898176 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:13:27.453905 140476350898176 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:13:27.570086 140476350898176 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:13:27.570354 140476350898176 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1908-8kgir/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:13:28.148177 140476350898176 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:13:28.262993 140476350898176 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:13:28.263258 140476350898176 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1908-8kgir
I0116 19:14:28.552073 140476350898176 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1908-8kgir deleted
I0116 19:14:28.552545 140476350898176 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0116 19:14:28.552820 140476350898176 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-05hlokk9:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1908-8kgir%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-05hlokk9%22;timeRange=2024-01-16T19:10:08.136670Z%2F2024-01-16T19:13:17.276855Z?project=grpc-testing
I0116 19:14:28.552976 140476350898176 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-bjsyu1nt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1908-8kgir%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-bjsyu1nt%22;timeRange=2024-01-16T19:08:57.037030Z%2F2024-01-16T19:14:28.552325Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
I0116 19:14:28.553201 140476350898176 base_testcase.py:52] ----- TestCase __main__.RoundRobinTest.test_round_robin PASSED -----
----------------------------------------------------------------------
Ran 1 test in 374.900s

OK
+ for test in "${test_suites[@]}"
+ run_test bootstrap_generator_test
+ local test_name=bootstrap_generator_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test'
+ set -x
+ python -m tests.bootstrap_generator_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723 --client_image=gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.6oxhUDkPHo/psm-interop/venv/bin/python
I0116 19:14:29.679081 139769193447424 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorClientTest -----
I0116 19:14:29.679201 139769193447424 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 19:14:29.679414 139769193447424 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 19:14:29.693211 139769193447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 19:14:29.706784 139769193447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0116 19:14:29.710362 139769193447424 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240116-1914-ho97x
I0116 19:14:30.058223 139769193447424 bootstrap_generator_testcase.py:88] Found unused xds port: 13631
I0116 19:14:30.058533 139769193447424 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1914-ho97x"
I0116 19:14:30.061291 139769193447424 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240116-1914-ho97x
type: GRPC
...

I0116 19:14:30.758355 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432470130-60f14f0921037-2f0bf7ff-144422dd
I0116 19:14:33.100929 139769193447424 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1914-ho97x"
I0116 19:14:33.106994 139769193447424 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1914-ho97x
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1914-ho97x
protocol: GRPC
...

I0116 19:14:34.389837 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432473109-60f14f0bf8288-0e17fd14-61c21a3a
I0116 19:14:49.445621 139769193447424 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1914-ho97x": psm-grpc-server:13631 -> psm-interop-backend-service-20240116-1914-ho97x
I0116 19:14:49.453682 139769193447424 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1914-ho97x
hostRules:
- hosts:
- psm-grpc-server:13631
pathMatcher: psm-interop-path-matcher-20240116-1914-ho97x
name: psm-interop-url-map-20240116-1914-ho97x
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1914-ho97x
name: psm-interop-path-matcher-20240116-1914-ho97x
...

I0116 19:14:50.100228 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432489456-60f14f1b8f2a1-101ae876-26fb5711
I0116 19:14:52.396907 139769193447424 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1914-ho97x proxy "GRPC" to URL map psm-interop-url-map-20240116-1914-ho97x
I0116 19:14:52.398716 139769193447424 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1914-ho97x
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1914-ho97x
validate_for_proxyless: true
...

I0116 19:14:53.219180 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432492400-60f14f1e5e005-696ba8b3-3e342f4a
I0116 19:14:55.439379 139769193447424 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1914-ho97x" in network "default-vpc": 0.0.0.0:13631 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1914-ho97x
I0116 19:14:55.441781 139769193447424 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1914-ho97x
network: global/networks/default-vpc
portRange: 13631
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1914-ho97x
...

I0116 19:14:56.135770 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432495443-60f14f2144f7d-3c610cf0-90f00c84
I0116 19:15:39.321905 139769193447424 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
I0116 19:15:39.402246 139769193447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1914-ho97x: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:15:39.402677 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1914-ho97x%22;timeRange=2024-01-16T19:15:39.402517Z%2F2024-01-16T19:45:39.402517Z?project=grpc-testing
I0116 19:15:39.406514 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1914-ho97x
labels:
name: psm-interop-server-20240116-1914-ho97x
owner: xds-k8s-interop-test
...

I0116 19:15:39.553241 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1914-ho97x created
I0116 19:15:39.555750 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1914-ho97x-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:15:39.618936 139769193447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:15:39.619180 139769193447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1914-ho97x
I0116 19:15:39.715922 139769193447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1914-ho97x: neg_name=psm-interop-server-20240116-1914-ho97x-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:15:39.716135 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1914-ho97x/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:15:40.374930 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:15:40.434257 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:15:40.442158 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1914-ho97x
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-z4dxp2l8
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-z4dxp2l8
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-z4dxp2l8
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1914-ho97x-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:15:40.509052 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:15:40.509312 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:15:40.610848 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-54f46cdcbc-6f7lb']
I0116 19:15:40.611136 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-54f46cdcbc-6f7lb to start
I0116 19:15:43.844061 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-server-54f46cdcbc-6f7lb ready, IP: 10.120.3.77
I0116 19:15:43.844356 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-54f46cdcbc-6f7lb to bootstrap_generator_test/test_app_logs/psm-interop-server-20240116-1914-ho97x_psm-grpc-server-54f46cdcbc-6f7lb.log
I0116 19:15:43.845026 139769100625472 k8s_log_collector.py:70] Starting log collection thread 139769100625472 for psm-grpc-server-54f46cdcbc-6f7lb
I0116 19:15:43.845249 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:15:50.184055 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:15:50.221134 139769193447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:15:50.484849 139769193447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1914-ho97x-psm-grpc-server" in zone us-central1-a
I0116 19:15:50.485090 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1914-ho97x: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1914-ho97x-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1914-ho97x-psm-grpc-server', zone='us-central1-a')}
I0116 19:15:50.488604 139769193447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1914-ho97x-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:15:51.395492 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432550490-60f14f55c42f7-83f0f5d1-a882e89e
I0116 19:16:42.809426 139769193447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1914-ho97x to report all backends healthy: ['psm-interop-server-20240116-1914-ho97x-psm-grpc-server']
I0116 19:16:55.525345 139769193447424 compute.py:494] Backend psm-interop-server-20240116-1914-ho97x-psm-grpc-server in zone us-central1-a reported healthy
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0116 19:16:59.530320 139769193447424 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.14.0 -----
I0116 19:16:59.532274 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1914-ho97x: server_target=xds:///psm-grpc-server:13631 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:16:59.532600 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1914-ho97x%22;timeRange=2024-01-16T19:16:59.532414Z%2F2024-01-16T19:46:59.532414Z?project=grpc-testing
I0116 19:16:59.534134 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1914-ho97x
labels:
name: psm-interop-client-20240116-1914-ho97x
owner: xds-k8s-interop-test
...

I0116 19:16:59.683568 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1914-ho97x created
I0116 19:16:59.683860 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:17:00.299826 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:17:00.362487 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:17:00.369554 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y6krsn70
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y6krsn70
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y6krsn70
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13631"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:17:00.445569 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:17:00.445826 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:17:00.548612 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5cf7f897f-n8wwk']
I0116 19:17:00.548907 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-5cf7f897f-n8wwk to start
I0116 19:17:03.789387 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-5cf7f897f-n8wwk ready, IP: 10.120.0.37
I0116 19:17:03.789669 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-5cf7f897f-n8wwk to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1914-ho97x_psm-grpc-client-5cf7f897f-n8wwk.log
I0116 19:17:03.790300 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-5cf7f897f-n8wwk
I0116 19:17:03.790703 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:17:03.960779 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:17:03.961342 139769193447424 client_app.py:244] [psm-grpc-client-5cf7f897f-n8wwk] Waiting to report a READY channel to xds:///psm-grpc-server:13631
I0116 19:17:04.067885 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:17:14.113101 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:17:24.158264 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:17:34.203951 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:17:44.248992 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:18:00.298588 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:18:25.358592 139769193447424 client_app.py:371] [psm-grpc-client-5cf7f897f-n8wwk] Server channel:
I0116 19:18:25.391230 139769193447424 client_app.py:384] [psm-grpc-client-5cf7f897f-n8wwk] Found subchannel in state READY:
I0116 19:18:25.391486 139769193447424 client_app.py:255] [psm-grpc-client-5cf7f897f-n8wwk] Channel to xds:///psm-grpc-server:13631 transitioned to state READY:
I0116 19:18:25.391689 139769193447424 grpc.py:75] [psm-grpc-client-5cf7f897f-n8wwk:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:18:25.426223 139769193447424 grpc.py:75] [psm-grpc-client-5cf7f897f-n8wwk:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:18:29.435467 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-5cf7f897f-n8wwk] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-54f46cdcbc-6f7lb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-54f46cdcbc-6f7lb: 100

I0116 19:18:29.435775 139769193447424 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0116 19:18:29.436156 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:18:39.585562 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:18:39.585885 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:18:40.243540 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:18:40.358030 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:18:40.358297 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1914-ho97x
I0116 19:19:10.564248 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1914-ho97x deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0116 19:19:10.564979 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0116 19:19:10.565254 139769193447424 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.13.0 -----
I0116 19:19:10.566948 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1914-ho97x: server_target=xds:///psm-grpc-server:13631 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:19:10.567147 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1914-ho97x%22;timeRange=2024-01-16T19:19:10.567044Z%2F2024-01-16T19:49:10.567044Z?project=grpc-testing
I0116 19:19:10.568669 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1914-ho97x
labels:
name: psm-interop-client-20240116-1914-ho97x
owner: xds-k8s-interop-test
...

I0116 19:19:10.628274 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1914-ho97x created
I0116 19:19:10.628572 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:19:11.197749 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:19:11.256613 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:19:11.263154 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3d8lzukm
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3d8lzukm
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3d8lzukm
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13631"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:19:11.333211 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:19:11.333466 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:19:11.436995 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-778bdc5f86-ktlsd']
I0116 19:19:11.437310 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-778bdc5f86-ktlsd to start
I0116 19:19:14.672634 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-778bdc5f86-ktlsd ready, IP: 10.120.0.38
I0116 19:19:14.672901 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-778bdc5f86-ktlsd to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1914-ho97x_psm-grpc-client-778bdc5f86-ktlsd.log
I0116 19:19:14.673239 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-778bdc5f86-ktlsd
I0116 19:19:14.673388 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:19:15.807188 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:19:15.807772 139769193447424 client_app.py:244] [psm-grpc-client-778bdc5f86-ktlsd] Waiting to report a READY channel to xds:///psm-grpc-server:13631
I0116 19:19:15.907651 139769193447424 client_app.py:371] [psm-grpc-client-778bdc5f86-ktlsd] Server channel:
I0116 19:19:15.940536 139769193447424 client_app.py:384] [psm-grpc-client-778bdc5f86-ktlsd] Found subchannel in state READY:
I0116 19:19:15.941004 139769193447424 client_app.py:255] [psm-grpc-client-778bdc5f86-ktlsd] Channel to xds:///psm-grpc-server:13631 transitioned to state READY:
I0116 19:19:15.941287 139769193447424 grpc.py:75] [psm-grpc-client-778bdc5f86-ktlsd:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:19:15.975398 139769193447424 grpc.py:75] [psm-grpc-client-778bdc5f86-ktlsd:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:19:19.991770 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-778bdc5f86-ktlsd] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-54f46cdcbc-6f7lb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-54f46cdcbc-6f7lb: 100

I0116 19:19:19.992037 139769193447424 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0116 19:19:19.992360 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:19:30.148285 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:19:30.148677 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:19:30.800735 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:19:30.919612 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:19:30.919932 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1914-ho97x
I0116 19:20:01.082922 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1914-ho97x deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0116 19:20:01.083522 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0116 19:20:01.083842 139769193447424 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.12.0 -----
I0116 19:20:01.085510 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1914-ho97x: server_target=xds:///psm-grpc-server:13631 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:20:01.085727 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1914-ho97x%22;timeRange=2024-01-16T19:20:01.085634Z%2F2024-01-16T19:50:01.085634Z?project=grpc-testing
I0116 19:20:01.087215 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1914-ho97x
labels:
name: psm-interop-client-20240116-1914-ho97x
owner: xds-k8s-interop-test
...

I0116 19:20:01.127996 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1914-ho97x created
I0116 19:20:01.128254 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:20:01.691637 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:20:01.731132 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:20:01.737546 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jczr9ecw
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jczr9ecw
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-jczr9ecw
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13631"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:20:01.785538 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:20:01.785781 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:20:01.867052 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6889f7f98c-b9h2c']
I0116 19:20:01.867319 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6889f7f98c-b9h2c to start
I0116 19:20:05.097701 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-6889f7f98c-b9h2c ready, IP: 10.120.0.39
I0116 19:20:05.097995 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6889f7f98c-b9h2c to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1914-ho97x_psm-grpc-client-6889f7f98c-b9h2c.log
I0116 19:20:05.098466 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-6889f7f98c-b9h2c
I0116 19:20:05.098720 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:20:06.217860 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:20:06.218438 139769193447424 client_app.py:244] [psm-grpc-client-6889f7f98c-b9h2c] Waiting to report a READY channel to xds:///psm-grpc-server:13631
I0116 19:20:06.318448 139769193447424 client_app.py:371] [psm-grpc-client-6889f7f98c-b9h2c] Server channel:
I0116 19:20:06.351348 139769193447424 client_app.py:384] [psm-grpc-client-6889f7f98c-b9h2c] Found subchannel in state READY:
I0116 19:20:06.351824 139769193447424 client_app.py:255] [psm-grpc-client-6889f7f98c-b9h2c] Channel to xds:///psm-grpc-server:13631 transitioned to state READY:
I0116 19:20:06.352101 139769193447424 grpc.py:75] [psm-grpc-client-6889f7f98c-b9h2c:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:20:06.386475 139769193447424 grpc.py:75] [psm-grpc-client-6889f7f98c-b9h2c:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:20:10.400862 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-6889f7f98c-b9h2c] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-54f46cdcbc-6f7lb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-54f46cdcbc-6f7lb: 100

I0116 19:20:10.401159 139769193447424 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0116 19:20:10.401549 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:20:20.530762 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:20:20.531079 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:20:21.067599 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:20:21.142073 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:20:21.142333 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1914-ho97x
I0116 19:20:51.281704 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1914-ho97x deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0116 19:20:51.282326 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0116 19:20:51.282580 139769193447424 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.11.0 -----
I0116 19:20:51.284133 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1914-ho97x: server_target=xds:///psm-grpc-server:13631 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:20:51.284358 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1914-ho97x%22;timeRange=2024-01-16T19:20:51.284267Z%2F2024-01-16T19:50:51.284267Z?project=grpc-testing
I0116 19:20:51.285902 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1914-ho97x
labels:
name: psm-interop-client-20240116-1914-ho97x
owner: xds-k8s-interop-test
...

I0116 19:20:51.325114 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1914-ho97x created
I0116 19:20:51.325371 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:20:51.996331 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:20:52.035141 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:20:52.042416 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1914-ho97x
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wtvh0hsg
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wtvh0hsg
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wtvh0hsg
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13631"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:20:52.091690 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:20:52.091938 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:20:52.173127 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7d58b9b9c5-8h4gz']
I0116 19:20:52.173406 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7d58b9b9c5-8h4gz to start
I0116 19:20:56.439652 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-7d58b9b9c5-8h4gz ready, IP: 10.120.0.40
I0116 19:20:56.439956 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7d58b9b9c5-8h4gz to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1914-ho97x_psm-grpc-client-7d58b9b9c5-8h4gz.log
I0116 19:20:56.440314 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-7d58b9b9c5-8h4gz
I0116 19:20:56.440485 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:20:56.518282 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:20:56.518907 139769193447424 client_app.py:244] [psm-grpc-client-7d58b9b9c5-8h4gz] Waiting to report a READY channel to xds:///psm-grpc-server:13631
I0116 19:20:56.620806 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:21:06.666787 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:21:16.712405 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:21:26.758519 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:21:36.800243 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:21:52.850266 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:22:17.911069 139769193447424 client_app.py:371] [psm-grpc-client-7d58b9b9c5-8h4gz] Server channel:
I0116 19:22:17.944944 139769193447424 client_app.py:384] [psm-grpc-client-7d58b9b9c5-8h4gz] Found subchannel in state READY:
I0116 19:22:17.945377 139769193447424 client_app.py:255] [psm-grpc-client-7d58b9b9c5-8h4gz] Channel to xds:///psm-grpc-server:13631 transitioned to state READY:
I0116 19:22:17.945738 139769193447424 grpc.py:75] [psm-grpc-client-7d58b9b9c5-8h4gz:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:22:17.981078 139769193447424 grpc.py:75] [psm-grpc-client-7d58b9b9c5-8h4gz:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:22:21.992171 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-7d58b9b9c5-8h4gz] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-54f46cdcbc-6f7lb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-54f46cdcbc-6f7lb: 100

I0116 19:22:21.992537 139769193447424 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0116 19:22:21.992903 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:22:32.178567 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:22:32.178917 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1914-ho97x/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:22:32.881948 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:22:33.002496 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:22:33.002743 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1914-ho97x
I0116 19:23:03.185330 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1914-ho97x deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0116 19:23:03.186004 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0116 19:23:03.242903 139769193447424 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:23:03.363753 139769193447424 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1914-ho97x-psm-grpc-server" in zone us-central1-a
I0116 19:23:03.364002 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1914-ho97x: set()
I0116 19:23:03.367044 139769193447424 compute.py:544] Patching compute resource:
---
backends: []
...

I0116 19:23:04.080660 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705432983369-60f150f297745-29b3a977-0b124c3c
I0116 19:23:49.413587 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:23:59.581551 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:23:59.581864 139769193447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:23:59.672305 139769193447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:23:59.672634 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1914-ho97x/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:24:00.321543 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:24:00.438272 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:24:00.438537 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1914-ho97x
I0116 19:25:00.713377 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1914-ho97x deleted
I0116 19:25:00.713664 139769193447424 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1914-ho97x"
I0116 19:25:01.401819 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433100717-60f1516280f46-b1611683-b3ba9c38
I0116 19:25:24.993204 139769193447424 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1914-ho97x"
I0116 19:25:25.100880 139769193447424 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1914-ho97x"
I0116 19:25:25.483558 139769193447424 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1914-ho97x"
I0116 19:25:25.937053 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433125487-60f1517a2026b-540f982c-e36dd8c2
I0116 19:25:28.214078 139769193447424 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1914-ho97x"
I0116 19:25:28.486281 139769193447424 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1914-ho97x"
I0116 19:25:29.095397 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433128491-60f1517cfd98d-705b9d09-fdc8f169
I0116 19:25:31.415767 139769193447424 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1914-ho97x"
I0116 19:25:31.951263 139769193447424 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1914-ho97x"
I0116 19:25:32.594960 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433131956-60f151804b83a-ecd21817-eabc5c63
I0116 19:25:47.741787 139769193447424 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1914-ho97x"
I0116 19:25:48.297115 139769193447424 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1914-ho97x"
I0116 19:25:48.654319 139769193447424 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1914-ho97x"
I0116 19:25:49.173916 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433148738-60f151904ca6a-3c1075cb-2430b933
I0116 19:25:53.671558 139769193447424 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorServerTest -----
I0116 19:25:53.671815 139769193447424 xds_k8s_testcase.py:158] Logs timezone: UTC
I0116 19:25:53.672528 139769193447424 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0116 19:25:53.686128 139769193447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0116 19:25:53.699385 139769193447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0116 19:25:53.700150 139769193447424 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240116-1925-mxn6y
I0116 19:25:54.125514 139769193447424 bootstrap_generator_testcase.py:88] Found unused xds port: 62941
I0116 19:25:54.125801 139769193447424 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240116-1925-mxn6y"
I0116 19:25:54.129369 139769193447424 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240116-1925-mxn6y
type: GRPC
...

I0116 19:25:54.736374 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433154170-60f151957af56-c1d6012d-7910212e
I0116 19:25:57.149591 139769193447424 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240116-1925-mxn6y"
I0116 19:25:57.155435 139769193447424 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240116-1925-mxn6y
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240116-1925-mxn6y
protocol: GRPC
...

I0116 19:25:58.441663 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433157158-60f15198544ec-7271ec2d-123ddf53
I0116 19:26:18.135216 139769193447424 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240116-1925-mxn6y": psm-grpc-server:62941 -> psm-interop-backend-service-20240116-1925-mxn6y
I0116 19:26:18.143027 139769193447424 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1925-mxn6y
hostRules:
- hosts:
- psm-grpc-server:62941
pathMatcher: psm-interop-path-matcher-20240116-1925-mxn6y
name: psm-interop-url-map-20240116-1925-mxn6y
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240116-1925-mxn6y
name: psm-interop-path-matcher-20240116-1925-mxn6y
...

I0116 19:26:18.797051 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433178145-60f151ac58199-818dba24-57574087
I0116 19:26:21.026382 139769193447424 traffic_director.py:578] Creating target psm-interop-target-proxy-20240116-1925-mxn6y proxy "GRPC" to URL map psm-interop-url-map-20240116-1925-mxn6y
I0116 19:26:21.028350 139769193447424 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240116-1925-mxn6y
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240116-1925-mxn6y
validate_for_proxyless: true
...

I0116 19:26:21.549543 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433181030-60f151af1898e-d11c1320-e533dac2
I0116 19:26:23.837231 139769193447424 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240116-1925-mxn6y" in network "default-vpc": 0.0.0.0:62941 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1925-mxn6y
I0116 19:26:23.839508 139769193447424 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240116-1925-mxn6y
network: global/networks/default-vpc
portRange: 62941
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240116-1925-mxn6y
...

I0116 19:26:24.496326 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433183841-60f151b1c6df6-7e922a9e-b0dfbc5f
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0116 19:27:12.135206 139769193447424 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.14.0 -----
I0116 19:27:12.221394 139769193447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1925-mxn6y: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:27:12.221830 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:27:12.221644Z%2F2024-01-16T19:57:12.221644Z?project=grpc-testing
I0116 19:27:12.223565 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1925-mxn6y
labels:
name: psm-interop-server-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:27:12.368769 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1925-mxn6y created
I0116 19:27:12.371333 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:27:12.439401 139769193447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:27:12.439664 139769193447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:27:12.530569 139769193447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1925-mxn6y: neg_name=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:27:12.530814 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:27:13.164348 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:27:13.221367 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:27:13.227601 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-s32q07re
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-s32q07re
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-s32q07re
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:27:13.294001 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:27:13.294235 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:27:13.397862 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5f787fbcb-h8q8g']
I0116 19:27:13.398151 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5f787fbcb-h8q8g to start
I0116 19:27:15.584053 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-server-5f787fbcb-h8q8g ready, IP: 10.120.4.155
I0116 19:27:15.584364 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5f787fbcb-h8q8g to bootstrap_generator_test/test_app_logs/psm-interop-server-20240116-1925-mxn6y_psm-grpc-server-5f787fbcb-h8q8g.log
I0116 19:27:15.584912 139769100625472 k8s_log_collector.py:70] Starting log collection thread 139769100625472 for psm-grpc-server-5f787fbcb-h8q8g
I0116 19:27:15.585155 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:27:22.968401 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:27:23.005240 139769193447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:27:23.277009 139769193447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:27:23.277239 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', zone='us-central1-a')}
I0116 19:27:23.280725 139769193447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:27:24.312025 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433243283-60f151ea76f58-4752c267-8f22b4a4
I0116 19:28:13.904132 139769193447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1925-mxn6y to report all backends healthy: ['psm-interop-server-20240116-1925-mxn6y-psm-grpc-server']
I0116 19:28:35.203360 139769193447424 compute.py:494] Backend psm-interop-server-20240116-1925-mxn6y-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:28:39.209849 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1925-mxn6y: server_target=xds:///psm-grpc-server:62941 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:28:39.210206 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:28:39.210071Z%2F2024-01-16T19:58:39.210071Z?project=grpc-testing
I0116 19:28:39.211754 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1925-mxn6y
labels:
name: psm-interop-client-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:28:39.259617 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1925-mxn6y created
I0116 19:28:39.259863 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:28:39.731913 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:28:39.799212 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:28:39.805616 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-f56jhjab
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-f56jhjab
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-f56jhjab
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:62941"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:28:39.880509 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:28:39.880732 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:28:40.005671 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7db57fd6fb-g29hv']
I0116 19:28:40.005942 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7db57fd6fb-g29hv to start
I0116 19:28:43.274352 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-7db57fd6fb-g29hv ready, IP: 10.120.0.41
I0116 19:28:43.274678 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7db57fd6fb-g29hv to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1925-mxn6y_psm-grpc-client-7db57fd6fb-g29hv.log
I0116 19:28:43.275195 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-7db57fd6fb-g29hv
I0116 19:28:43.275409 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:28:43.459298 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:28:43.459875 139769193447424 client_app.py:244] [psm-grpc-client-7db57fd6fb-g29hv] Waiting to report a READY channel to xds:///psm-grpc-server:62941
I0116 19:28:43.560826 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:28:53.606134 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:29:03.652059 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:29:13.695762 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:29:23.740618 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:29:39.790179 139769193447424 client_app.py:371] [psm-grpc-client-7db57fd6fb-g29hv] Server channel:
I0116 19:29:39.823308 139769193447424 client_app.py:384] [psm-grpc-client-7db57fd6fb-g29hv] Found subchannel in state READY:
I0116 19:29:39.823658 139769193447424 client_app.py:255] [psm-grpc-client-7db57fd6fb-g29hv] Channel to xds:///psm-grpc-server:62941 transitioned to state READY:
I0116 19:29:39.823943 139769193447424 grpc.py:75] [psm-grpc-client-7db57fd6fb-g29hv:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:29:39.858213 139769193447424 grpc.py:75] [psm-grpc-client-7db57fd6fb-g29hv:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:29:43.853656 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-7db57fd6fb-g29hv] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f787fbcb-h8q8g: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5f787fbcb-h8q8g: 100

I0116 19:29:43.853942 139769193447424 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0116 19:29:43.854306 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:29:54.066877 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:29:54.067179 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:29:54.676720 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:29:54.813958 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:29:54.814213 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1925-mxn6y
I0116 19:30:25.020892 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1925-mxn6y deleted
I0116 19:30:25.086319 139769193447424 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:30:25.354521 139769193447424 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:30:25.354836 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: set()
I0116 19:30:25.358036 139769193447424 compute.py:544] Patching compute resource:
---
backends: []
...

I0116 19:30:26.081158 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433425360-60f152981b79c-933a77a9-fc7fa252
I0116 19:31:04.845146 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:31:15.038203 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:31:15.038490 139769193447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:31:15.147992 139769193447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:31:15.148271 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:31:15.729983 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:31:15.856011 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:31:15.856266 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:33:17.203626 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1925-mxn6y deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0116 19:33:17.204137 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0116 19:33:17.204337 139769193447424 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.13.0 -----
I0116 19:33:17.206177 139769193447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1925-mxn6y: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:33:17.206425 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:33:17.206325Z%2F2024-01-16T20:03:17.206325Z?project=grpc-testing
I0116 19:33:17.207997 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1925-mxn6y
labels:
name: psm-interop-server-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:33:17.266218 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1925-mxn6y created
I0116 19:33:17.269052 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:33:17.336273 139769193447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:33:17.336599 139769193447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:33:17.434475 139769193447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1925-mxn6y: neg_name=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:33:17.434699 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:33:17.949727 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:33:18.007815 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:33:18.014202 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hy1z8z5p
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hy1z8z5p
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hy1z8z5p
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:33:18.082262 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:33:18.082514 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:33:18.185654 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5479749879-f8qws']
I0116 19:33:18.185964 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5479749879-f8qws to start
I0116 19:33:20.379011 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-server-5479749879-f8qws ready, IP: 10.120.4.156
I0116 19:33:20.379342 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5479749879-f8qws to bootstrap_generator_test/test_app_logs/psm-interop-server-20240116-1925-mxn6y_psm-grpc-server-5479749879-f8qws.log
I0116 19:33:20.379917 139769100625472 k8s_log_collector.py:70] Starting log collection thread 139769100625472 for psm-grpc-server-5479749879-f8qws
I0116 19:33:20.380322 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:33:27.756256 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:33:27.793572 139769193447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:33:27.915807 139769193447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:33:27.916067 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', zone='us-central1-a')}
I0116 19:33:27.919858 139769193447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:33:28.782103 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433607921-60f1534636123-559a7785-8923da3c
I0116 19:34:20.619820 139769193447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1925-mxn6y to report all backends healthy: ['psm-interop-server-20240116-1925-mxn6y-psm-grpc-server']
I0116 19:34:29.222434 139769193447424 compute.py:494] Backend psm-interop-server-20240116-1925-mxn6y-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:34:33.229027 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1925-mxn6y: server_target=xds:///psm-grpc-server:62941 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:34:33.229386 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:34:33.229271Z%2F2024-01-16T20:04:33.229271Z?project=grpc-testing
I0116 19:34:33.230997 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1925-mxn6y
labels:
name: psm-interop-client-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:34:33.273659 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1925-mxn6y created
I0116 19:34:33.273901 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:34:33.905771 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:34:33.958439 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:34:33.965052 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xwozdda3
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xwozdda3
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xwozdda3
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:62941"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:34:34.035647 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:34:34.035922 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:34:34.143844 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-69f86dd7b4-rg9fb']
I0116 19:34:34.144124 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-69f86dd7b4-rg9fb to start
I0116 19:34:37.372637 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-69f86dd7b4-rg9fb ready, IP: 10.120.3.79
I0116 19:34:37.372932 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-69f86dd7b4-rg9fb to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1925-mxn6y_psm-grpc-client-69f86dd7b4-rg9fb.log
I0116 19:34:37.373401 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-69f86dd7b4-rg9fb
I0116 19:34:37.373776 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:34:38.510689 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:34:38.511227 139769193447424 client_app.py:244] [psm-grpc-client-69f86dd7b4-rg9fb] Waiting to report a READY channel to xds:///psm-grpc-server:62941
I0116 19:34:38.610484 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:34:48.655839 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:34:58.700901 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:35:08.746143 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:35:18.790817 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:35:34.840717 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:35:59.900981 139769193447424 client_app.py:371] [psm-grpc-client-69f86dd7b4-rg9fb] Server channel:
I0116 19:35:59.933776 139769193447424 client_app.py:384] [psm-grpc-client-69f86dd7b4-rg9fb] Found subchannel in state READY:
I0116 19:35:59.934120 139769193447424 client_app.py:255] [psm-grpc-client-69f86dd7b4-rg9fb] Channel to xds:///psm-grpc-server:62941 transitioned to state READY:
I0116 19:35:59.934432 139769193447424 grpc.py:75] [psm-grpc-client-69f86dd7b4-rg9fb:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:35:59.968770 139769193447424 grpc.py:75] [psm-grpc-client-69f86dd7b4-rg9fb:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:36:03.983061 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-69f86dd7b4-rg9fb] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5479749879-f8qws: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5479749879-f8qws: 100

I0116 19:36:03.983421 139769193447424 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0116 19:36:03.983829 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:36:14.180579 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:36:14.180986 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:36:14.753622 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:36:14.860326 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:36:14.860611 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1925-mxn6y
I0116 19:36:45.041314 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1925-mxn6y deleted
I0116 19:36:45.096744 139769193447424 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:36:45.356858 139769193447424 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:36:45.357147 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: set()
I0116 19:36:45.360611 139769193447424 compute.py:544] Patching compute resource:
---
backends: []
...

I0116 19:36:46.136004 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705433805363-60f154028173b-0e74e2e4-6af684e1
I0116 19:37:37.702477 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:37:47.891226 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:37:47.891545 139769193447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:37:47.988350 139769193447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:37:47.988641 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:37:48.601554 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:37:48.719907 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:37:48.720160 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:39:49.121311 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1925-mxn6y deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0116 19:39:49.121930 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0116 19:39:49.122201 139769193447424 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.12.0 -----
I0116 19:39:49.123929 139769193447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1925-mxn6y: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:39:49.124146 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:39:49.124042Z%2F2024-01-16T20:09:49.124042Z?project=grpc-testing
I0116 19:39:49.125766 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1925-mxn6y
labels:
name: psm-interop-server-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:39:49.186597 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1925-mxn6y created
I0116 19:39:49.189304 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:39:49.259890 139769193447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:39:49.260136 139769193447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:39:49.356208 139769193447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1925-mxn6y: neg_name=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:39:49.356508 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:39:50.003172 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:39:50.058793 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:39:50.065344 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ctfhw8r0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ctfhw8r0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ctfhw8r0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:39:50.130673 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:39:50.130941 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:39:50.234348 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-55676b5d9c-b54gh']
I0116 19:39:50.234657 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-55676b5d9c-b54gh to start
I0116 19:39:53.471081 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-server-55676b5d9c-b54gh ready, IP: 10.120.4.157
I0116 19:39:53.471358 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-55676b5d9c-b54gh to bootstrap_generator_test/test_app_logs/psm-interop-server-20240116-1925-mxn6y_psm-grpc-server-55676b5d9c-b54gh.log
I0116 19:39:53.471884 139769100625472 k8s_log_collector.py:70] Starting log collection thread 139769100625472 for psm-grpc-server-55676b5d9c-b54gh
I0116 19:39:53.472081 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:40:02.926162 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:40:02.962985 139769193447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:40:03.235701 139769193447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:40:03.235954 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', zone='us-central1-a')}
I0116 19:40:03.239504 139769193447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:40:04.129284 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434003242-60f154bf37cc0-7ca9643d-2cbd8609
I0116 19:40:49.392418 139769193447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1925-mxn6y to report all backends healthy: ['psm-interop-server-20240116-1925-mxn6y-psm-grpc-server']
I0116 19:40:49.723915 139769193447424 compute.py:494] Backend psm-interop-server-20240116-1925-mxn6y-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:40:53.730950 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1925-mxn6y: server_target=xds:///psm-grpc-server:62941 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:40:53.731300 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:40:53.731190Z%2F2024-01-16T20:10:53.731190Z?project=grpc-testing
I0116 19:40:53.732981 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1925-mxn6y
labels:
name: psm-interop-client-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:40:53.808608 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1925-mxn6y created
I0116 19:40:53.808851 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:40:54.395308 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:40:54.458014 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:40:54.464342 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zyv5ha11
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zyv5ha11
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zyv5ha11
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:62941"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:40:54.534836 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:40:54.535106 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:40:54.635735 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-58465f4f-jb4cx']
I0116 19:40:54.636030 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-58465f4f-jb4cx to start
I0116 19:40:57.870450 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-58465f4f-jb4cx ready, IP: 10.120.3.80
I0116 19:40:57.870757 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-58465f4f-jb4cx to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1925-mxn6y_psm-grpc-client-58465f4f-jb4cx.log
I0116 19:40:57.871306 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-58465f4f-jb4cx
I0116 19:40:57.871552 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:40:59.083801 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:40:59.084335 139769193447424 client_app.py:244] [psm-grpc-client-58465f4f-jb4cx] Waiting to report a READY channel to xds:///psm-grpc-server:62941
I0116 19:40:59.182431 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:41:09.227601 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:41:19.272177 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:41:29.316906 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:41:39.361759 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:41:55.410956 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:42:20.471344 139769193447424 client_app.py:371] [psm-grpc-client-58465f4f-jb4cx] Server channel:
I0116 19:42:20.503751 139769193447424 client_app.py:384] [psm-grpc-client-58465f4f-jb4cx] Found subchannel in state READY:
I0116 19:42:20.504135 139769193447424 client_app.py:255] [psm-grpc-client-58465f4f-jb4cx] Channel to xds:///psm-grpc-server:62941 transitioned to state READY:
I0116 19:42:20.504428 139769193447424 grpc.py:75] [psm-grpc-client-58465f4f-jb4cx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:42:20.538494 139769193447424 grpc.py:75] [psm-grpc-client-58465f4f-jb4cx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:42:24.566661 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-58465f4f-jb4cx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-55676b5d9c-b54gh: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-55676b5d9c-b54gh: 100

I0116 19:42:24.566945 139769193447424 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0116 19:42:24.567313 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:42:34.719357 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:42:34.719683 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:42:35.239394 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:42:35.361404 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:42:35.361675 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1925-mxn6y
I0116 19:43:05.576652 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1925-mxn6y deleted
I0116 19:43:05.630568 139769193447424 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:43:05.897243 139769193447424 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:43:05.897513 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: set()
I0116 19:43:05.900483 139769193447424 compute.py:544] Patching compute resource:
---
backends: []
...

I0116 19:43:06.589274 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434185902-60f1556d6ab30-b1300e39-52ab5a77
I0116 19:43:51.885504 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:44:02.048845 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:44:02.049193 139769193447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:44:02.151029 139769193447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:44:02.151314 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:44:02.650446 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:44:02.757743 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:44:02.758019 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:46:03.172620 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1925-mxn6y deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0116 19:46:03.173316 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0116 19:46:03.173580 139769193447424 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.11.0 -----
I0116 19:46:03.175306 139769193447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240116-1925-mxn6y: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0116 19:46:03.175514 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:46:03.175408Z%2F2024-01-16T20:16:03.175408Z?project=grpc-testing
I0116 19:46:03.177107 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240116-1925-mxn6y
labels:
name: psm-interop-server-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:46:03.237860 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240116-1925-mxn6y created
I0116 19:46:03.240668 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0116 19:46:03.310837 139769193447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0116 19:46:03.311102 139769193447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:46:03.400324 139769193447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240116-1925-mxn6y: neg_name=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server, port=8080, zones=['us-central1-a']
I0116 19:46:03.400623 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:46:03.930797 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:46:03.983013 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0116 19:46:03.989428 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240116-1925-mxn6y
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-9m0nn6z2
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-9m0nn6z2
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-9m0nn6z2
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240116-1925-mxn6y-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:46:04.055993 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0116 19:46:04.056254 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0116 19:46:04.157402 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-777d58694f-h72cw']
I0116 19:46:04.157665 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-777d58694f-h72cw to start
I0116 19:46:07.388077 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-server-777d58694f-h72cw ready, IP: 10.120.0.44
I0116 19:46:07.388362 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-777d58694f-h72cw to bootstrap_generator_test/test_app_logs/psm-interop-server-20240116-1925-mxn6y_psm-grpc-server-777d58694f-h72cw.log
I0116 19:46:07.388909 139769100625472 k8s_log_collector.py:70] Starting log collection thread 139769100625472 for psm-grpc-server-777d58694f-h72cw
I0116 19:46:07.389239 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0116 19:46:13.719626 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0116 19:46:13.756609 139769193447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:46:13.879564 139769193447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:46:13.879842 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server', zone='us-central1-a')}
I0116 19:46:13.883525 139769193447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240116-1925-mxn6y-psm-grpc-server
maxRatePerEndpoint: 5
...

I0116 19:46:14.792432 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434373886-60f15620b106a-432699a8-e67e64e8
I0116 19:47:00.096607 139769193447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240116-1925-mxn6y to report all backends healthy: ['psm-interop-server-20240116-1925-mxn6y-psm-grpc-server']
I0116 19:47:00.312889 139769193447424 compute.py:494] Backend psm-interop-server-20240116-1925-mxn6y-psm-grpc-server in zone us-central1-a reported healthy
I0116 19:47:04.319935 139769193447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240116-1925-mxn6y: server_target=xds:///psm-grpc-server:62941 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0116 19:47:04.320310 139769193447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240116-1925-mxn6y%22;timeRange=2024-01-16T19:47:04.320188Z%2F2024-01-16T20:17:04.320188Z?project=grpc-testing
I0116 19:47:04.322001 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240116-1925-mxn6y
labels:
name: psm-interop-client-20240116-1925-mxn6y
owner: xds-k8s-interop-test
...

I0116 19:47:04.364206 139769193447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240116-1925-mxn6y created
I0116 19:47:04.364520 139769193447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:47:04.967135 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0116 19:47:05.034779 139769193447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0116 19:47:05.041495 139769193447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240116-1925-mxn6y
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-o32vflpe
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-o32vflpe
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-o32vflpe
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:eb521e502093ca03aea6f558edf5fa36a7caf723
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:62941"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0116 19:47:05.106989 139769193447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0116 19:47:05.107223 139769193447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0116 19:47:05.221784 139769193447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5dc8769855-kjc6b']
I0116 19:47:05.222054 139769193447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-5dc8769855-kjc6b to start
I0116 19:47:08.452223 139769193447424 k8s_base_runner.py:820] Pod psm-grpc-client-5dc8769855-kjc6b ready, IP: 10.120.2.158
I0116 19:47:08.452568 139769193447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-5dc8769855-kjc6b to bootstrap_generator_test/test_app_logs/psm-interop-client-20240116-1925-mxn6y_psm-grpc-client-5dc8769855-kjc6b.log
I0116 19:47:08.453057 139769092232768 k8s_log_collector.py:70] Starting log collection thread 139769092232768 for psm-grpc-client-5dc8769855-kjc6b
I0116 19:47:08.453268 139769193447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0116 19:47:09.587081 139769193447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0116 19:47:09.587640 139769193447424 client_app.py:244] [psm-grpc-client-5dc8769855-kjc6b] Waiting to report a READY channel to xds:///psm-grpc-server:62941
I0116 19:47:09.686220 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:47:19.731518 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:47:29.776189 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:47:39.821407 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:47:49.866477 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:48:05.915588 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:48:30.975731 139769193447424 client_app.py:371] [psm-grpc-client-5dc8769855-kjc6b] Server channel:
I0116 19:48:31.008220 139769193447424 client_app.py:384] [psm-grpc-client-5dc8769855-kjc6b] Found subchannel in state READY:
I0116 19:48:31.008537 139769193447424 client_app.py:255] [psm-grpc-client-5dc8769855-kjc6b] Channel to xds:///psm-grpc-server:62941 transitioned to state READY:
I0116 19:48:31.008762 139769193447424 grpc.py:75] [psm-grpc-client-5dc8769855-kjc6b:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0116 19:48:31.042643 139769193447424 grpc.py:75] [psm-grpc-client-5dc8769855-kjc6b:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0116 19:48:35.054515 139769193447424 xds_k8s_testcase.py:601] [psm-grpc-client-5dc8769855-kjc6b] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-777d58694f-h72cw: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-777d58694f-h72cw: 100

I0116 19:48:35.054872 139769193447424 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0116 19:48:35.055272 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0116 19:48:45.238528 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0116 19:48:45.238878 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240116-1925-mxn6y/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:48:45.827199 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0116 19:48:45.946580 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0116 19:48:45.946852 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240116-1925-mxn6y
I0116 19:49:16.097364 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240116-1925-mxn6y deleted
I0116 19:49:16.154119 139769193447424 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0116 19:49:16.263529 139769193447424 traffic_director.py:268] Loaded NEG "psm-interop-server-20240116-1925-mxn6y-psm-grpc-server" in zone us-central1-a
I0116 19:49:16.263787 139769193447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240116-1925-mxn6y: set()
I0116 19:49:16.266702 139769193447424 compute.py:544] Patching compute resource:
---
backends: []
...

I0116 19:49:16.985607 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434556268-60f156cea00af-ee506e85-18af3a8a
I0116 19:49:45.131437 139769193447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0116 19:49:55.297228 139769193447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0116 19:49:55.297548 139769193447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0116 19:49:55.391248 139769193447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0116 19:49:55.391520 139769193447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240116-1925-mxn6y/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0116 19:49:55.978266 139769193447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0116 19:49:56.092613 139769193447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0116 19:49:56.092875 139769193447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240116-1925-mxn6y
I0116 19:52:26.602885 139769193447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240116-1925-mxn6y deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0116 19:52:26.603502 139769193447424 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0116 19:52:26.603670 139769193447424 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240116-1925-mxn6y"
I0116 19:52:27.373581 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434746607-60f15784258f3-57a6df2c-4fa054ab
I0116 19:52:46.655818 139769193447424 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240116-1925-mxn6y"
I0116 19:52:46.741034 139769193447424 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240116-1925-mxn6y"
I0116 19:52:47.185325 139769193447424 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240116-1925-mxn6y"
I0116 19:52:47.698575 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434767188-60f15797c614c-3d1fcb35-4e56161c
I0116 19:52:49.977822 139769193447424 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240116-1925-mxn6y"
I0116 19:52:50.198335 139769193447424 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240116-1925-mxn6y"
I0116 19:52:50.793070 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434770203-60f1579aa6334-0cbfbfe5-bd420242
I0116 19:52:53.072624 139769193447424 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240116-1925-mxn6y"
I0116 19:52:53.464527 139769193447424 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240116-1925-mxn6y"
I0116 19:52:54.073635 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434773469-60f1579dc3a1c-eeb803c9-06d693bd
I0116 19:53:26.329304 139769193447424 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240116-1925-mxn6y"
I0116 19:53:26.791607 139769193447424 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240116-1925-mxn6y"
I0116 19:53:27.198967 139769193447424 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240116-1925-mxn6y"
I0116 19:53:27.692737 139769193447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705434807263-60f157bdfe10e-ee6751ee-629c770d
----------------------------------------------------------------------
Ran 8 tests in 2342.431s

OK
+ echo 'Failed test suites: 0'
Failed test suites: 0


[ID: 6142015] Command finished after 6869 secs, exit value: 0


Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[11:53:32 PST] Collecting build artifacts from build VM
[11:53:33 PST] Kokoro builder finished