Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[11:44:17] Transferring environment variable script to build VM
[11:44:18] Transferring kokoro_log_reader.py to build VM
[11:44:19] Transferring source code to build VM
[11:44:21] Executing build script on build VM



[ID: 4859929] Executing command via SSH:
export KOKORO_BUILD_NUMBER="541"
export KOKORO_JOB_NAME="grpc/go/master/branch/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-go/test/kokoro/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-go/test/kokoro/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/grpc/master/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-go
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.2 LTS
Release: 22.04
Codename: jammy
Python 3.10.6
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,1
GIT_ORIGIN_URL,https://github.com/dfawley/grpc-go.git
GIT_COMMIT_SHORT,0af3cad6
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128497 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128442 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 128042 files and directories currently installed.)
Preparing to unpack .../00-python3.10-dev_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-dev (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../01-libpython3.10-dev_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-dev:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../02-libpython3.10_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../03-python3.10_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../04-libpython3.10-stdlib_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-stdlib:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../05-python3.10-minimal_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-minimal (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Preparing to unpack .../06-libpython3.10-minimal_3.10.12-1~22.04.2_amd64.deb ...
Unpacking libpython3.10-minimal:amd64 (3.10.12-1~22.04.2) over (3.10.6-1~22.04.2ubuntu1) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
Preparing to unpack .../07-google-cloud-sdk-gke-gcloud-auth-plugin_442.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (442.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../08-kubectl_1.27.4-00_amd64.deb ...
Unpacking kubectl (1.27.4-00) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../09-python3-pip-whl_22.0.2+dfsg-1ubuntu0.3_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.3) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../10-python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../11-python3.10-venv_3.10.12-1~22.04.2_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.2) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.3) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (442.0.0-0) ...
Setting up kubectl (1.27.4-00) ...
Setting up libpython3.10-minimal:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10-minimal (3.10.12-1~22.04.2) ...
Setting up libpython3.10-stdlib:amd64 (3.10.12-1~22.04.2) ...
Setting up libpython3.10:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10 (3.10.12-1~22.04.2) ...
Setting up libpython3.10-dev:amd64 (3.10.12-1~22.04.2) ...
Setting up python3.10-dev (3.10.12-1~22.04.2) ...
Setting up python3.10-venv (3.10.12-1~22.04.2) ...
Processing triggers for man-db (2.10.2-1) ...
Processing triggers for libc-bin (2.35-0ubuntu3.1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/grpc/grpc.git branch master to /tmp/tmp.VtPSjI0WCh/grpc
Cloning into '/tmp/tmp.VtPSjI0WCh/grpc'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 2.1 MB/s eta 0:00:00
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 15.0 MB/s eta 0:00:00
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 27.6 MB/s eta 0:00:00
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 13.6 MB/s eta 0:00:00
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 21.7 MB/s eta 0:00:00
Collecting grpcio==1.51.1 (from -r requirements.lock (line 6))
Downloading grpcio-1.51.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 72.9 MB/s eta 0:00:00
Collecting grpcio-health-checking==1.48.2 (from -r requirements.lock (line 7))
Downloading grpcio_health_checking-1.48.2-py3-none-any.whl (18 kB)
Collecting grpcio-tools==1.48.2 (from -r requirements.lock (line 8))
Downloading grpcio_tools-1.48.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.4/2.4 MB 103.6 MB/s eta 0:00:00
Collecting grpcio-channelz==1.48.2 (from -r requirements.lock (line 9))
Downloading grpcio_channelz-1.48.2-py3-none-any.whl (21 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 10))
Obtaining dependency information for kubernetes==27.2.0 from https://files.pythonhosted.org/packages/99/89/3ab0cb3069f49ae2eaf73f884c82164f18f70fcc598e0312edea71614ad7/kubernetes-27.2.0-py2.py3-none-any.whl.metadata
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 11))
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 12))
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==21.3 (from -r requirements.lock (line 13))
Downloading packaging-21.3-py3-none-any.whl (40 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 8.8 MB/s eta 0:00:00
Collecting Pygments==2.14.0 (from -r requirements.lock (line 14))
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 88.1 MB/s eta 0:00:00
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 15))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 46.3 MB/s eta 0:00:00
Collecting protobuf==3.20.3 (from -r requirements.lock (line 16))
Downloading protobuf-3.20.3-cp310-cp310-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 81.6 MB/s eta 0:00:00
Collecting xds-protos==0.0.11 (from -r requirements.lock (line 17))
Downloading xds_protos-0.0.11-py3-none-any.whl (1.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.8/1.8 MB 96.9 MB/s eta 0:00:00
Collecting cachetools==5.3.0 (from -r requirements.lock (line 19))
Downloading cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting certifi==2022.12.7 (from -r requirements.lock (line 20))
Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 31.8 MB/s eta 0:00:00
Collecting charset-normalizer==3.0.1 (from -r requirements.lock (line 21))
Downloading charset_normalizer-3.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 198.8/198.8 kB 33.1 MB/s eta 0:00:00
Collecting google-api-core==2.11.0 (from -r requirements.lock (line 22))
Downloading google_api_core-2.11.0-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.3/120.3 kB 25.0 MB/s eta 0:00:00
Collecting google-auth==2.16.0 (from -r requirements.lock (line 23))
Downloading google_auth-2.16.0-py2.py3-none-any.whl (177 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 177.8/177.8 kB 31.7 MB/s eta 0:00:00
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 24))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.58.0 (from -r requirements.lock (line 25))
Downloading googleapis_common_protos-1.58.0-py2.py3-none-any.whl (223 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.0/223.0 kB 37.6 MB/s eta 0:00:00
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 26))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.48.2 (from -r requirements.lock (line 27))
Downloading grpcio_status-1.48.2-py3-none-any.whl (14 kB)
Collecting httplib2==0.21.0 (from -r requirements.lock (line 28))
Downloading httplib2-0.21.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.8/96.8 kB 21.6 MB/s eta 0:00:00
Collecting idna==3.4 (from -r requirements.lock (line 29))
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 13.5 MB/s eta 0:00:00
Collecting MarkupSafe==2.1.2 (from -r requirements.lock (line 30))
Downloading MarkupSafe-2.1.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 31))
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 28.6 MB/s eta 0:00:00
Collecting proto-plus==1.22.2 (from -r requirements.lock (line 32))
Downloading proto_plus-1.22.2-py3-none-any.whl (47 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.9/47.9 kB 8.9 MB/s eta 0:00:00
Collecting pyasn1==0.4.8 (from -r requirements.lock (line 33))
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 16.4 MB/s eta 0:00:00
Collecting pyasn1-modules==0.2.8 (from -r requirements.lock (line 34))
Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 31.7 MB/s eta 0:00:00
Collecting pyparsing==3.0.9 (from -r requirements.lock (line 35))
Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 18.4 MB/s eta 0:00:00
Collecting requests==2.28.2 (from -r requirements.lock (line 36))
Downloading requests-2.28.2-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 13.8 MB/s eta 0:00:00
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 37))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 38))
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 39))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.14 (from -r requirements.lock (line 40))
Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 29.9 MB/s eta 0:00:00
Collecting websocket-client==1.5.1 (from -r requirements.lock (line 41))
Downloading websocket_client-1.5.1-py3-none-any.whl (55 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.9/55.9 kB 9.4 MB/s eta 0:00:00
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0 (from google-cloud-secret-manager==2.15.1->-r requirements.lock (line 5))
Obtaining dependency information for google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0 from https://files.pythonhosted.org/packages/6e/c4/c3cd048b6cbeba8d9ae50dd7643ac065b85237338aa7501b0efae91eb4d9/google_api_core-2.11.1-py3-none-any.whl.metadata
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.48.2->-r requirements.lock (line 8)) (68.0.0)
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0 (from grpc-google-iam-v1==0.12.6->-r requirements.lock (line 26))
Obtaining dependency information for googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0 from https://files.pythonhosted.org/packages/a7/bc/416a1ffeba4dcd072bc10523dac9ed97f2e7fc4b760580e2bdbdc1e2afdd/googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
INFO: pip is looking at multiple versions of google-api-core[grpc] to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of googleapis-common-protos[grpc] to determine which version is compatible with other requirements. This could take a while.
Obtaining dependency information for googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0 from https://files.pythonhosted.org/packages/b3/b7/bbaa556e9ff0580f408c64ccf4db0c1414eec79e7151d33a10bc209ffb6d/googleapis_common_protos-1.59.1-py2.py3-none-any.whl.metadata
Downloading googleapis_common_protos-1.59.1-py2.py3-none-any.whl.metadata (1.5 kB)
Downloading googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.6/223.6 kB 35.4 MB/s eta 0:00:00
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 89.0 MB/s eta 0:00:00
Installing collected packages: pyasn1, charset-normalizer, websocket-client, urllib3, uritemplate, six, rsa, PyYAML, pyparsing, Pygments, pyasn1-modules, protobuf, oauthlib, MarkupSafe, idna, grpcio, certifi, cachetools, xds-protos, tenacity, requests, python-dateutil, proto-plus, packaging, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, google-auth, absl-py, requests-oauthlib, grpcio-status, google-auth-httplib2, google-api-core, kubernetes, grpc-google-iam-v1, google-api-python-client, google-cloud-secret-manager
Successfully installed Mako-1.2.4 MarkupSafe-2.1.2 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.0.1 google-api-core-2.11.0 google-api-python-client-1.12.11 google-auth-2.16.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.58.0 grpc-google-iam-v1-0.12.6 grpcio-1.51.1 grpcio-channelz-1.48.2 grpcio-health-checking-1.48.2 grpcio-status-1.48.2 grpcio-tools-1.48.2 httplib2-0.21.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-21.3 proto-plus-1.22.2 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 python-dateutil-2.8.2 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.14 websocket-client-1.5.1 xds-protos-0.0.11
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.0
certifi 2022.12.7
charset-normalizer 3.0.1
google-api-core 2.11.0
google-api-python-client 1.12.11
google-auth 2.16.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.58.0
grpc-google-iam-v1 0.12.6
grpcio 1.51.1
grpcio-channelz 1.48.2
grpcio-health-checking 1.48.2
grpcio-status 1.48.2
grpcio-tools 1.48.2
httplib2 0.21.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.2
oauthlib 3.2.2
packaging 21.3
pip 23.2.1
proto-plus 1.22.2
protobuf 3.20.3
pyasn1 0.4.8
pyasn1-modules 0.2.8
Pygments 2.14.0
pyparsing 3.0.9
python-dateutil 2.8.2
PyYAML 6.0
requests 2.28.2
requests-oauthlib 1.3.1
rsa 4.9
setuptools 68.0.0
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.14
websocket-client 1.5.1
xds-protos 0.0.11
Generate python code from grpc.testing protos: src/proto/grpc/testing/test.proto src/proto/grpc/testing/messages.proto src/proto/grpc/testing/empty.proto
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing:
total 84
-rw-rw-r-- 1 kbuilder kbuilder 1163 Aug 11 18:45 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Aug 11 18:45 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 32763 Aug 11 18:45 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Aug 11 18:45 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 3934 Aug 11 18:45 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 34950 Aug 11 18:45 test_pb2_grpc.py
Server image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
Server image not found
Client image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
Client image not found
Building Go xDS interop test app Docker images
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.51kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.6s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.6s

#5 [stage-1 1/2] FROM docker.io/library/alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#5 resolve docker.io/library/alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a 0.0s done
#5 sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a 1.64kB / 1.64kB done
#5 sha256:c5c5fda71656f28e49ac9c5416b3643eaa6a108a8093151d6d1afc9463be8e33 528B / 528B done
#5 sha256:7e01a0d0a1dcd9e539f8e9bbd80106d59efbdf97293b3d38f5d7a34501526cdb 1.47kB / 1.47kB done
#5 sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de 0B / 3.40MB 0.1s
#5 extracting sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de
#5 sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de 3.40MB / 3.40MB 0.2s done
#5 DONE 0.4s

#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:445f34008a77b0b98bf1821bf7ef5e37bb63cc42d22ee7c21cc17041070d134f
#6 resolve docker.io/library/golang:1.21-alpine@sha256:445f34008a77b0b98bf1821bf7ef5e37bb63cc42d22ee7c21cc17041070d134f 0.0s done
#6 sha256:dd8888bb7f1b0b05e1e625aa29483f50f38a9b64073a4db00b04076cec52b71c 1.16kB / 1.16kB done
#6 sha256:c9e410d017b6e59e5c2c06f67b44cc13020b698b35b572edf9c94288220bc172 6.16kB / 6.16kB done
#6 sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de 3.40MB / 3.40MB 0.2s done
#6 sha256:c4d48a809fc2256f8aa0aeee47998488d64409855adba00a7cb3007ab9f3286e 284.69kB / 284.69kB 0.2s done
#6 sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 29.36MB / 66.88MB 0.4s
#6 sha256:445f34008a77b0b98bf1821bf7ef5e37bb63cc42d22ee7c21cc17041070d134f 1.65kB / 1.65kB done
#6 sha256:ad3a456e5733f4fd4d5c3f67fcf931e1034a03ab86e308ca9e8cc62249ecf768 155B / 155B 0.2s done
#6 extracting sha256:7264a8db6415046d36d16ba98b79778e18accee6ffa71850405994cffa9be7de 0.1s done
#6 extracting sha256:c4d48a809fc2256f8aa0aeee47998488d64409855adba00a7cb3007ab9f3286e
#6 sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 45.09MB / 66.88MB 0.5s
#6 extracting sha256:c4d48a809fc2256f8aa0aeee47998488d64409855adba00a7cb3007ab9f3286e 0.1s done
#6 sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 60.82MB / 66.88MB 0.6s
#6 sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 66.88MB / 66.88MB 0.7s
#6 ...

#7 [internal] load build context
#7 transferring context: 76.82MB 0.7s done
#7 DONE 0.8s

#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:445f34008a77b0b98bf1821bf7ef5e37bb63cc42d22ee7c21cc17041070d134f
#6 sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 66.88MB / 66.88MB 0.8s done
#6 extracting sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d
#6 extracting sha256:4f1c88b9dad58987453186064cc54e131c5ec4b47f021c054e3d218e3e0f758d 2.2s done
#6 extracting sha256:ad3a456e5733f4fd4d5c3f67fcf931e1034a03ab86e308ca9e8cc62249ecf768 done
#6 DONE 3.4s

#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 DONE 1.6s

#9 [build 3/4] COPY . .
#9 DONE 0.4s

#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/client/client.go
#10 0.354 go: downloading google.golang.org/protobuf v1.31.0
#10 0.355 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98
#10 0.370 go: downloading github.com/golang/protobuf v1.5.3
#10 0.381 go: downloading golang.org/x/net v0.12.0
#10 0.437 go: downloading google.golang.org/genproto v0.0.0-20230711160842-782d3b101e98
#10 0.490 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.520 go: downloading golang.org/x/sys v0.10.0
#10 0.839 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.840 go: downloading github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4
#10 0.917 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.315 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.316 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20230711160842-782d3b101e98
#10 1.318 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.319 go: downloading golang.org/x/oauth2 v0.10.0
#10 1.319 go: downloading golang.org/x/text v0.11.0
#10 1.320 go: downloading golang.org/x/sync v0.3.0
#10 1.409 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.435 go: downloading cloud.google.com/go/compute v1.21.0
#10 DONE 18.7s

#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/client .
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:3182b27da4bfb4dc04c3bdf67c8b459d95251af8c807f10a69368bc8c8de301d done
#12 naming to gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 done
#12 DONE 0.2s
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.51kB done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.1s

#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.1s

#5 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:445f34008a77b0b98bf1821bf7ef5e37bb63cc42d22ee7c21cc17041070d134f
#5 DONE 0.0s

#6 [stage-1 1/2] FROM docker.io/library/alpine@sha256:7144f7bab3d4c2648d7e59409f15ec52a18006a128c733fcff20d3a4a54ba44a
#6 CACHED

#7 [internal] load build context
#7 transferring context: 73.44kB 0.0s done
#7 DONE 0.1s

#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 CACHED

#9 [build 3/4] COPY . .
#9 CACHED

#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/server/server.go
#10 0.339 go: downloading google.golang.org/protobuf v1.31.0
#10 0.345 go: downloading github.com/golang/protobuf v1.5.3
#10 0.354 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98
#10 0.371 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.421 go: downloading google.golang.org/genproto v0.0.0-20230711160842-782d3b101e98
#10 0.477 go: downloading golang.org/x/net v0.12.0
#10 0.488 go: downloading golang.org/x/sys v0.10.0
#10 0.807 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.816 go: downloading github.com/cncf/xds/go v0.0.0-20230607035331-e9ce68804cb4
#10 0.881 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.283 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.283 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20230711160842-782d3b101e98
#10 1.288 go: downloading golang.org/x/oauth2 v0.10.0
#10 1.289 go: downloading golang.org/x/text v0.11.0
#10 1.290 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.292 go: downloading golang.org/x/sync v0.3.0
#10 1.364 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.381 go: downloading cloud.google.com/go/compute v1.21.0
#10 DONE 18.5s

#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/server .
#11 DONE 0.1s

#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:d862b52eb6ff10675d2ae06ed04e370fac207e833caf879bad63d9eb53198f41 done
#12 naming to gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 done
#12 DONE 0.2s
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-client]
5d0d3ed5e6d5: Preparing
4693057ce236: Preparing
4693057ce236: Layer already exists
5d0d3ed5e6d5: Pushed
0af3cad6ea5703485ec781cf0a15b4ae412010a4: digest: sha256:d1443224fd721832e29757a42ef5f0d07b4b3a1b10d932e469ae4c197425252e size: 740
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-server]
04a81e642750: Preparing
4693057ce236: Preparing
4693057ce236: Layer already exists
04a81e642750: Pushed
0af3cad6ea5703485ec781cf0a15b4ae412010a4: digest: sha256:12bda3f41bf9ebf8dab9ddbae14336ceda0b6276a5fe79a6e44faaa9a4fe0930 size: 740
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test'
+ python -m tests.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 18:46:18.429384 140417864421376 xds_k8s_testcase.py:130] ----- Testing AffinityTest -----
I0811 18:46:18.429526 140417864421376 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 18:46:18.429780 140417864421376 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 18:46:19.361347 140417864421376 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 18:46:19.377433 140417864421376 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0811 18:46:19.381940 140417864421376 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1846-a7duo
I0811 18:46:19.739357 140417864421376 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test
I0811 18:46:20.118758 140417864421376 xds_k8s_testcase.py:625] Found unused xds port: 18228
I0811 18:46:20.119158 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0811 18:46:20.119297 140417864421376 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1846-a7duo"
I0811 18:46:20.122476 140417864421376 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1846-a7duo
type: GRPC
...

I0811 18:46:20.723364 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779580177-602aa20ea2c71-145258b6-0c136f9c
I0811 18:46:23.074289 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0811 18:46:23.074538 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0811 18:46:23.074635 140417864421376 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1846-a7duo"
I0811 18:46:23.082100 140417864421376 compute.py:530] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1846-a7duo
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20230811-1846-a7duo
protocol: GRPC
sessionAffinity: HEADER_FIELD
...

I0811 18:46:23.966497 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779583084-602aa211688c4-a1a976d9-19fce95a
I0811 18:46:45.883828 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0811 18:46:45.884071 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0811 18:46:45.884165 140417864421376 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1846-a7duo": psm-grpc-server:18228 -> psm-interop-backend-service-20230811-1846-a7duo
I0811 18:46:45.894824 140417864421376 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1846-a7duo
hostRules:
- hosts:
- psm-grpc-server:18228
pathMatcher: psm-interop-path-matcher-20230811-1846-a7duo
name: psm-interop-url-map-20230811-1846-a7duo
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1846-a7duo
name: psm-interop-path-matcher-20230811-1846-a7duo
...

I0811 18:46:46.693198 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779605897-602aa22729f54-fa5ef48f-037f91cb
I0811 18:46:49.146164 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0811 18:46:49.146407 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0811 18:46:49.146510 140417864421376 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1846-a7duo proxy "GRPC" to URL map psm-interop-url-map-20230811-1846-a7duo
I0811 18:46:49.148375 140417864421376 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1846-a7duo
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1846-a7duo
validate_for_proxyless: true
...

I0811 18:46:50.034448 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779609151-602aa22a446f8-e1e8c203-2c8084ae
I0811 18:46:52.308883 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0811 18:46:52.309139 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0811 18:46:52.309246 140417864421376 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1846-a7duo" in network "default-vpc": 0.0.0.0:18228 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1846-a7duo
I0811 18:46:52.311752 140417864421376 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1846-a7duo
network: global/networks/default-vpc
portRange: 18228
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1846-a7duo
...

I0811 18:46:53.282662 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779612314-602aa22d48b59-2fcfa61a-a5e275c8
I0811 18:47:06.495749 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0811 18:47:06.495991 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0811 18:47:06.496116 140417864421376 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1846-a7duo: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0811 18:47:06.496380 140417864421376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1846-a7duo%22;timeRange=2023-08-11T18:47:06.496202Z%2F2023-08-11T19:17:06.496202Z?project=grpc-testing
I0811 18:47:06.501273 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1846-a7duo
labels:
name: psm-interop-server-20230811-1846-a7duo
owner: xds-k8s-interop-test
...

I0811 18:47:06.672209 140417864421376 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1846-a7duo created
I0811 18:47:06.675165 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1846-a7duo
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1846-a7duo-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 18:47:06.748408 140417864421376 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 18:47:06.748636 140417864421376 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 18:47:06.848894 140417864421376 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1846-a7duo-psm-grpc-server in zones=['us-central1-a']
I0811 18:47:06.849102 140417864421376 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1846-a7duo/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:47:07.830599 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1846-a7duo
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 18:47:07.892172 140417864421376 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 18:47:07.900248 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1846-a7duo
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ropkk8h6
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ropkk8h6
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ropkk8h6
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1846-a7duo-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 18:47:07.975422 140417864421376 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 18:47:07.975671 140417864421376 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0811 18:47:08.137708 140417864421376 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-555b4c7457-lxctw', 'psm-grpc-server-555b4c7457-vldlx', 'psm-grpc-server-555b4c7457-xlbsj']
I0811 18:47:08.138114 140417864421376 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-555b4c7457-lxctw to start
I0811 18:47:11.376044 140417864421376 k8s_base_runner.py:554] Pod psm-grpc-server-555b4c7457-lxctw ready, IP: 10.120.17.227
I0811 18:47:11.376301 140417864421376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-555b4c7457-lxctw to affinity_test/test_app_logs/psm-interop-server-20230811-1846-a7duo_psm-grpc-server-555b4c7457-lxctw.log
I0811 18:47:11.376746 140417773856320 k8s_log_collector.py:70] Starting log collection thread 140417773856320 for psm-grpc-server-555b4c7457-lxctw
I0811 18:47:11.376887 140417864421376 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-555b4c7457-vldlx to start
I0811 18:47:11.484189 140417864421376 k8s_base_runner.py:554] Pod psm-grpc-server-555b4c7457-vldlx ready, IP: 10.120.9.222
I0811 18:47:11.484437 140417864421376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-555b4c7457-vldlx to affinity_test/test_app_logs/psm-interop-server-20230811-1846-a7duo_psm-grpc-server-555b4c7457-vldlx.log
I0811 18:47:11.484872 140417765463616 k8s_log_collector.py:70] Starting log collection thread 140417765463616 for psm-grpc-server-555b4c7457-vldlx
I0811 18:47:11.485047 140417864421376 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-555b4c7457-xlbsj to start
I0811 18:47:11.659242 140417864421376 k8s_base_runner.py:554] Pod psm-grpc-server-555b4c7457-xlbsj ready, IP: 10.120.11.231
I0811 18:47:11.659499 140417864421376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-555b4c7457-xlbsj to affinity_test/test_app_logs/psm-interop-server-20230811-1846-a7duo_psm-grpc-server-555b4c7457-xlbsj.log
I0811 18:47:11.659855 140417757070912 k8s_log_collector.py:70] Starting log collection thread 140417757070912 for psm-grpc-server-555b4c7457-xlbsj
I0811 18:47:11.659988 140417864421376 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0811 18:47:17.031916 140417864421376 k8s_base_runner.py:522] Deployment psm-grpc-server has 3 replicas available
I0811 18:47:17.032621 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0811 18:47:17.032742 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0811 18:47:17.069531 140417864421376 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 18:47:17.187874 140417864421376 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1846-a7duo-psm-grpc-server" in zone us-central1-a
I0811 18:47:17.188082 140417864421376 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1846-a7duo: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1846-a7duo-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1846-a7duo-psm-grpc-server', zone='us-central1-a')}
I0811 18:47:17.191970 140417864421376 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1846-a7duo-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 18:47:18.419777 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779637194-602aa24502f69-1f62a025-f0811575
I0811 18:48:16.066558 140417864421376 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1846-a7duo to report all backends healthy: ['psm-interop-server-20230811-1846-a7duo-psm-grpc-server']
I0811 18:48:24.791310 140417864421376 compute.py:487] Backend psm-interop-server-20230811-1846-a7duo-psm-grpc-server in zone us-central1-a reported healthy
I0811 18:48:28.795999 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0811 18:48:28.796234 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0811 18:48:28.796355 140417864421376 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1846-a7duo: server_target=xds:///psm-grpc-server:18228 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0811 18:48:28.796587 140417864421376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1846-a7duo%22;timeRange=2023-08-11T18:48:28.796444Z%2F2023-08-11T19:18:28.796444Z?project=grpc-testing
I0811 18:48:28.798531 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1846-a7duo
labels:
name: psm-interop-client-20230811-1846-a7duo
owner: xds-k8s-interop-test
...

I0811 18:48:28.840091 140417864421376 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1846-a7duo created
I0811 18:48:28.840344 140417864421376 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1846-a7duo/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:48:29.535356 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1846-a7duo
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 18:48:29.598003 140417864421376 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 18:48:29.606260 140417864421376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1846-a7duo
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wzq2jj4k
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wzq2jj4k
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wzq2jj4k
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:18228"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 18:48:29.682826 140417864421376 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 18:48:29.683059 140417864421376 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 18:48:29.789182 140417864421376 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-54779fc6c9-bwrtm']
I0811 18:48:29.789473 140417864421376 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-54779fc6c9-bwrtm to start
I0811 18:48:34.069971 140417864421376 k8s_base_runner.py:554] Pod psm-grpc-client-54779fc6c9-bwrtm ready, IP: 10.120.10.247
I0811 18:48:34.070330 140417864421376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-54779fc6c9-bwrtm to affinity_test/test_app_logs/psm-interop-client-20230811-1846-a7duo_psm-grpc-client-54779fc6c9-bwrtm.log
I0811 18:48:34.070785 140417748678208 k8s_log_collector.py:70] Starting log collection thread 140417748678208 for psm-grpc-client-54779fc6c9-bwrtm
I0811 18:48:34.070972 140417864421376 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 18:48:37.296655 140417864421376 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 18:48:37.297256 140417864421376 client_app.py:200] [psm-grpc-client-54779fc6c9-bwrtm] Waiting to report a READY channel to xds:///psm-grpc-server:18228
I0811 18:48:37.397879 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:48:47.442789 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:48:57.487784 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:49:07.532917 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:49:17.578577 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:49:33.627660 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:49:58.687733 140417864421376 client_app.py:236] [psm-grpc-client-54779fc6c9-bwrtm] Server channel:
I0811 18:49:58.783613 140417864421376 client_app.py:249] [psm-grpc-client-54779fc6c9-bwrtm] Found subchannel in state READY:
I0811 18:49:58.783917 140417864421376 client_app.py:211] [psm-grpc-client-54779fc6c9-bwrtm] Channel to xds:///psm-grpc-server:18228 transitioned to state READY:
I0811 18:49:58.784156 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:49:58.818192 140417864421376 affinity_test.py:108] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/edca7b8b-5110-4871-a78e-0cd043f493e9",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.10.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.58.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230811-1846-a7duo"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230811-1846-a7duo",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-08-11T18:49:58.387870325Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"versionInfo": "2",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3042556283019017412_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.227",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.9.222",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.11.231",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-08-11T18:49:54.376406664Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:18228",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:18228",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-08-11T18:49:58.388064266Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:18228"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228-route-0"
}
]
}
]
},
"lastUpdated": "2023-08-11T18:49:58.388186105Z",
"clientStatus": "ACKED"
}
]
}
I0811 18:49:58.818979 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0811 18:49:58.819090 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0811 18:49:58.819198 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:49:58.852630 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0811 18:49:58.852860 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0811 18:49:58.853094 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:50:02.850991 140417864421376 xds_k8s_testcase.py:558] [psm-grpc-client-54779fc6c9-bwrtm] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-555b4c7457-lxctw: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-555b4c7457-lxctw: 100

I0811 18:50:02.851247 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0811 18:50:02.851360 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0811 18:50:02.851490 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:50:06.850980 140417864421376 xds_k8s_testcase.py:558] [psm-grpc-client-54779fc6c9-bwrtm] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-555b4c7457-lxctw: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-555b4c7457-lxctw: 100

I0811 18:50:06.981170 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0811 18:50:06.981415 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0811 18:50:06.981515 140417864421376 server_app.py:98] [psm-grpc-server-555b4c7457-lxctw] >> Setting health status to NOT_SERVING
I0811 18:50:06.981846 140417864421376 grpc.py:74] [psm-grpc-server-555b4c7457-lxctw:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 18:50:07.080652 140417864421376 grpc.py:74] [psm-grpc-server-555b4c7457-lxctw:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 18:50:07.113537 140417864421376 server_app.py:102] [psm-grpc-server-555b4c7457-lxctw] << Health status status: NOT_SERVING

I0811 18:50:07.113763 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0811 18:50:07.113849 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0811 18:50:07.113969 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:50:07.147575 140417864421376 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0811 18:50:17.158030 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:50:17.193174 140417864421376 affinity_test.py:185] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/edca7b8b-5110-4871-a78e-0cd043f493e9",
"cluster": "cluster",
"metadata": {
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384",
"INSTANCE_IP": "10.120.10.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.58.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:18228"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228-route-0"
}
]
}
]
},
"lastUpdated": "2023-08-11T18:49:58.388186105Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230811-1846-a7duo"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230811-1846-a7duo",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-08-11T18:49:58.387870325Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"versionInfo": "5",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_8321429818356481808",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_3042556283019017412_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.17.227",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.9.222",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.11.231",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-08-11T18:50:15.377103406Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:18228",
"versionInfo": "1691779788214929928",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:18228",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230811-1846-a7duo_0_psm-grpc-server:18228"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-08-11T18:49:58.388064266Z",
"clientStatus": "ACKED"
}
]
}
I0811 18:50:17.193878 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0811 18:50:17.193985 140417864421376 xds_k8s_testcase.py:239] --- Starting subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0811 18:50:17.194109 140417864421376 grpc.py:74] [psm-grpc-client-54779fc6c9-bwrtm:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:50:21.210168 140417864421376 xds_k8s_testcase.py:558] [psm-grpc-client-54779fc6c9-bwrtm] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-555b4c7457-vldlx: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-555b4c7457-vldlx: 100

I0811 18:50:21.210521 140417864421376 xds_k8s_testcase.py:244] --- Finished subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0811 18:50:21.210716 140417864421376 xds_k8s_testcase.py:640] ----- TestMethod __main__.AffinityTest.test_affinity teardown -----
I0811 18:50:21.485336 140417864421376 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1846-a7duo"
I0811 18:50:22.273701 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779821489-602aa2f4c4c24-63b17561-f16617af
I0811 18:50:35.682971 140417864421376 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1846-a7duo"
I0811 18:50:35.844598 140417864421376 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:50:35.844817 140417864421376 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1846-a7duo"
I0811 18:50:36.288103 140417864421376 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:50:36.288366 140417864421376 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1846-a7duo"
I0811 18:50:36.886291 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779836291-602aa302e29fe-8535ee59-5cd391fd
I0811 18:50:39.267921 140417864421376 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1846-a7duo"
I0811 18:50:39.538250 140417864421376 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:50:39.538491 140417864421376 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1846-a7duo"
I0811 18:50:40.896253 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779839544-602aa305fccd4-dd4021c7-03f3f96a
I0811 18:50:43.392277 140417864421376 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1846-a7duo"
I0811 18:50:43.848462 140417864421376 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:50:43.848682 140417864421376 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1846-a7duo"
I0811 18:50:44.587446 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779843853-602aa30a18c4b-246b0b88-b1b3dec3
I0811 18:51:19.566417 140417864421376 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1846-a7duo"
I0811 18:51:20.131085 140417864421376 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:51:20.131339 140417864421376 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1846-a7duo"
I0811 18:51:20.674234 140417864421376 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1846-a7duo" not deleted since it does not exist
I0811 18:51:20.674477 140417864421376 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1846-a7duo"
I0811 18:51:21.258874 140417864421376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691779880766-602aa32d4cc74-ace76b15-572e1923
I0811 18:51:25.817006 140417864421376 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 18:51:36.007879 140417864421376 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1846-a7duo/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:51:36.846091 140417864421376 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 18:51:36.968225 140417864421376 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1846-a7duo
I0811 18:52:07.158164 140417864421376 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 18:52:17.325932 140417864421376 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 18:52:17.444771 140417864421376 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1846-a7duo/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:52:18.105980 140417864421376 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 18:52:18.228646 140417864421376 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1846-a7duo
I0811 18:54:48.763357 140417864421376 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 18:54:48.763794 140417864421376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-wzq2jj4k:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1846-a7duo%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-wzq2jj4k%22;timeRange=2023-08-11T18:48:28.796436Z%2F2023-08-11T18:52:07.152094Z?project=grpc-testing
I0811 18:54:48.763986 140417864421376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-ropkk8h6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1846-a7duo%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-ropkk8h6%22;timeRange=2023-08-11T18:47:06.496195Z%2F2023-08-11T18:54:48.763193Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
----------------------------------------------------------------------
Ran 1 test in 510.338s

OK
+ for test in "${test_suites[@]}"
+ run_test api_listener_test
+ local test_name=api_listener_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test'
+ set -x
+ python -m tests.api_listener_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 18:54:49.754614 139712558514176 xds_k8s_testcase.py:130] ----- Testing ApiListenerTest -----
I0811 18:54:49.754751 139712558514176 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 18:54:49.754986 139712558514176 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 18:54:49.770819 139712558514176 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 18:54:49.785888 139712558514176 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0811 18:54:49.790183 139712558514176 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1854-1msg0
I0811 18:54:50.168691 139712558514176 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
I0811 18:54:50.460794 139712558514176 xds_k8s_testcase.py:625] Found unused xds port: 25330
I0811 18:54:50.461139 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0811 18:54:50.461219 139712558514176 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1854-1msg0"
I0811 18:54:50.464299 139712558514176 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1854-1msg0
type: GRPC
...

I0811 18:54:51.110636 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780090527-602aa3f557da1-dc0011b0-1da7aea7
I0811 18:54:53.349127 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0811 18:54:53.349289 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0811 18:54:53.349357 139712558514176 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1854-1msg0"
I0811 18:54:53.356328 139712558514176 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1854-1msg0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1854-1msg0
protocol: GRPC
...

I0811 18:54:54.085930 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780093358-602aa3f80b225-f0a509bd-6e80a7fc
I0811 18:55:22.650556 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0811 18:55:22.650771 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0811 18:55:22.650859 139712558514176 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1854-1msg0": psm-grpc-server:25330 -> psm-interop-backend-service-20230811-1854-1msg0
I0811 18:55:22.661768 139712558514176 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1854-1msg0
hostRules:
- hosts:
- psm-grpc-server:25330
pathMatcher: psm-interop-path-matcher-20230811-1854-1msg0
name: psm-interop-url-map-20230811-1854-1msg0
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1854-1msg0
name: psm-interop-path-matcher-20230811-1854-1msg0
...

I0811 18:55:23.190903 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780122663-602aa413fdae3-13396089-0a52e9b1
I0811 18:55:25.466737 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0811 18:55:25.466970 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0811 18:55:25.467065 139712558514176 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1854-1msg0 proxy "GRPC" to URL map psm-interop-url-map-20230811-1854-1msg0
I0811 18:55:25.469004 139712558514176 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1854-1msg0
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1854-1msg0
validate_for_proxyless: true
...

I0811 18:55:25.961565 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780125471-602aa416ab2c2-b3f5cc5a-ca31883d
I0811 18:55:28.261843 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0811 18:55:28.262041 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0811 18:55:28.262139 139712558514176 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1854-1msg0" in network "default-vpc": 0.0.0.0:25330 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1854-1msg0
I0811 18:55:28.264616 139712558514176 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1854-1msg0
network: global/networks/default-vpc
portRange: 25330
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1854-1msg0
...

I0811 18:55:29.196514 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780128266-602aa41955aec-79392621-816110ba
I0811 18:55:42.172617 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0811 18:55:42.172833 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0811 18:55:42.172926 139712558514176 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1854-1msg0: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 18:55:42.173194 139712558514176 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1854-1msg0%22;timeRange=2023-08-11T18:55:42.173044Z%2F2023-08-11T19:25:42.173044Z?project=grpc-testing
I0811 18:55:42.178020 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1854-1msg0
labels:
name: psm-interop-server-20230811-1854-1msg0
owner: xds-k8s-interop-test
...

I0811 18:55:42.313558 139712558514176 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1854-1msg0 created
I0811 18:55:42.316408 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1854-1msg0
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1854-1msg0-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 18:55:42.393322 139712558514176 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 18:55:42.393522 139712558514176 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 18:55:42.495197 139712558514176 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1854-1msg0-psm-grpc-server in zones=['us-central1-a']
I0811 18:55:42.495396 139712558514176 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1854-1msg0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:55:43.515844 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1854-1msg0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 18:55:43.572105 139712558514176 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 18:55:43.580464 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1854-1msg0
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ggqbwmxx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ggqbwmxx
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ggqbwmxx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1854-1msg0-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 18:55:43.657895 139712558514176 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 18:55:43.658117 139712558514176 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 18:55:43.762772 139712558514176 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5878cccbb8-2q88c']
I0811 18:55:43.763019 139712558514176 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5878cccbb8-2q88c to start
I0811 18:55:47.000168 139712558514176 k8s_base_runner.py:554] Pod psm-grpc-server-5878cccbb8-2q88c ready, IP: 10.120.11.232
I0811 18:55:47.000402 139712558514176 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5878cccbb8-2q88c to api_listener_test/test_app_logs/psm-interop-server-20230811-1854-1msg0_psm-grpc-server-5878cccbb8-2q88c.log
I0811 18:55:47.000824 139712467609152 k8s_log_collector.py:70] Starting log collection thread 139712467609152 for psm-grpc-server-5878cccbb8-2q88c
I0811 18:55:47.000962 139712558514176 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 18:55:55.479983 139712558514176 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 18:55:55.480373 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0811 18:55:55.480479 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0811 18:55:55.516062 139712558514176 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 18:55:55.780007 139712558514176 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1854-1msg0-psm-grpc-server" in zone us-central1-a
I0811 18:55:55.780223 139712558514176 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1854-1msg0: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1854-1msg0-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1854-1msg0-psm-grpc-server', zone='us-central1-a')}
I0811 18:55:55.784105 139712558514176 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1854-1msg0-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 18:55:56.860840 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780155786-602aa433943f8-88849a82-df168181
I0811 18:56:57.834928 139712558514176 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1854-1msg0 to report all backends healthy: ['psm-interop-server-20230811-1854-1msg0-psm-grpc-server']
I0811 18:57:10.736393 139712558514176 compute.py:487] Backend psm-interop-server-20230811-1854-1msg0-psm-grpc-server in zone us-central1-a reported healthy
I0811 18:57:14.741193 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0811 18:57:14.741433 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0811 18:57:14.741534 139712558514176 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1854-1msg0: server_target=xds:///psm-grpc-server:25330 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 18:57:14.741744 139712558514176 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1854-1msg0%22;timeRange=2023-08-11T18:57:14.741609Z%2F2023-08-11T19:27:14.741609Z?project=grpc-testing
I0811 18:57:14.743644 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1854-1msg0
labels:
name: psm-interop-client-20230811-1854-1msg0
owner: xds-k8s-interop-test
...

I0811 18:57:14.829318 139712558514176 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1854-1msg0 created
I0811 18:57:14.829573 139712558514176 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1854-1msg0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 18:57:15.565956 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1854-1msg0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 18:57:15.629946 139712558514176 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 18:57:15.637962 139712558514176 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1854-1msg0
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-agmgx7a6
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-agmgx7a6
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-agmgx7a6
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:25330"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 18:57:15.712202 139712558514176 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 18:57:15.712456 139712558514176 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 18:57:15.816682 139712558514176 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6fdbbf769b-6r2ck']
I0811 18:57:15.816987 139712558514176 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-6fdbbf769b-6r2ck to start
I0811 18:57:18.010944 139712558514176 k8s_base_runner.py:554] Pod psm-grpc-client-6fdbbf769b-6r2ck ready, IP: 10.120.17.228
I0811 18:57:18.011195 139712558514176 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-6fdbbf769b-6r2ck to api_listener_test/test_app_logs/psm-interop-client-20230811-1854-1msg0_psm-grpc-client-6fdbbf769b-6r2ck.log
I0811 18:57:18.011612 139712459216448 k8s_log_collector.py:70] Starting log collection thread 139712459216448 for psm-grpc-client-6fdbbf769b-6r2ck
I0811 18:57:18.011744 139712558514176 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 18:57:20.264341 139712558514176 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 18:57:20.264905 139712558514176 client_app.py:200] [psm-grpc-client-6fdbbf769b-6r2ck] Waiting to report a READY channel to xds:///psm-grpc-server:25330
I0811 18:57:20.367202 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:57:30.412392 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:57:40.457084 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:57:50.502516 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:58:00.546915 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:58:16.596370 139712558514176 client_app.py:236] [psm-grpc-client-6fdbbf769b-6r2ck] Server channel:
I0811 18:58:16.629153 139712558514176 client_app.py:249] [psm-grpc-client-6fdbbf769b-6r2ck] Found subchannel in state READY:
I0811 18:58:16.629457 139712558514176 client_app.py:211] [psm-grpc-client-6fdbbf769b-6r2ck] Channel to xds:///psm-grpc-server:25330 transitioned to state READY:
I0811 18:58:16.629595 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0811 18:58:16.629672 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0811 18:58:16.629868 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:58:16.664683 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0811 18:58:16.664886 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0811 18:58:16.665122 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:58:20.678835 139712558514176 xds_k8s_testcase.py:558] [psm-grpc-client-6fdbbf769b-6r2ck] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5878cccbb8-2q88c: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5878cccbb8-2q88c: 100

I0811 18:58:20.679100 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0811 18:58:20.679197 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0811 18:58:20.679285 139712558514176 traffic_director.py:536] Creating alternative URL map "psm-interop-url-map-alt-20230811-1854-1msg0": psm-grpc-server:25330 -> psm-interop-backend-service-20230811-1854-1msg0
I0811 18:58:20.684354 139712558514176 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1854-1msg0
hostRules:
- hosts:
- psm-grpc-server:25330
pathMatcher: psm-interop-path-matcher-20230811-1854-1msg0
name: psm-interop-url-map-alt-20230811-1854-1msg0
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1854-1msg0
name: psm-interop-path-matcher-20230811-1854-1msg0
...

I0811 18:58:21.192096 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780300686-602aa4bdc4512-36c95bfd-33457f01
I0811 18:58:23.629082 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0811 18:58:23.629308 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0811 18:58:23.629404 139712558514176 traffic_director.py:609] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20230811-1854-1msg0" to URL map psm-interop-url-map-alt-20230811-1854-1msg0
I0811 18:58:23.631155 139712558514176 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20230811-1854-1msg0
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20230811-1854-1msg0
validate_for_proxyless: false
...

I0811 18:58:24.330692 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780303633-602aa4c093c32-c58e3c80-7352197e
I0811 18:58:26.667924 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0811 18:58:26.668128 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0811 18:58:26.668224 139712558514176 traffic_director.py:679] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20230811-1854-1msg0" in network "default-vpc": 10.10.10.10:25330 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20230811-1854-1msg0
I0811 18:58:26.670809 139712558514176 compute.py:530] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20230811-1854-1msg0
network: global/networks/default-vpc
portRange: 25330
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20230811-1854-1msg0
...

I0811 18:58:27.523962 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780306672-602aa4c379cff-c5e8e333-60ba86cf
I0811 18:58:41.173079 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0811 18:58:41.173329 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0811 18:58:41.173507 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:58:45.198820 139712558514176 xds_k8s_testcase.py:558] [psm-grpc-client-6fdbbf769b-6r2ck] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5878cccbb8-2q88c: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5878cccbb8-2q88c: 100

I0811 18:58:45.199141 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:58:45.232862 139712558514176 api_listener_test.py:110] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/284b5334-3a65-42af-9409-90fefdfff60b",
"cluster": "cluster",
"metadata": {
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384",
"INSTANCE_IP": "10.120.17.228",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.58.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:25330",
"versionInfo": "1691780306320794402",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:25330",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20230811-1854-1msg0_0_psm-grpc-server:25330"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2023-08-11T18:58:35.703137286Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1854-1msg0_0_psm-grpc-server:25330",
"versionInfo": "1691780306320794402",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1854-1msg0_0_psm-grpc-server:25330",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:25330"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_1156848340499643666",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20230811-1854-1msg0_0_psm-grpc-server:25330-route-0"
}
]
}
]
},
"lastUpdated": "2023-08-11T18:58:35.703256106Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_1156848340499643666",
"versionInfo": "1691780306320794402",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_1156848340499643666",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20230811-1854-1msg0"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20230811-1854-1msg0",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2023-08-11T18:58:35.702971726Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_1156848340499643666",
"versionInfo": "5",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_1156848340499643666",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_9090546456912935616_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.11.232",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2023-08-11T18:58:41.425975513Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1691780306320794402
I0811 18:58:45.233550 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0811 18:58:45.233658 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0811 18:58:45.233727 139712558514176 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1854-1msg0"
I0811 18:58:46.148013 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780325237-602aa4d52e29c-33d661c8-f8e05ce7
I0811 18:58:59.343567 139712558514176 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1854-1msg0"
I0811 18:58:59.823095 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780339346-602aa4e2a2c01-c45560c4-9bab9ee9
I0811 18:59:02.227341 139712558514176 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1854-1msg0"
I0811 18:59:02.950947 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780342233-602aa4e56397b-338627a1-3f35d553
I0811 18:59:05.196151 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0811 18:59:05.196397 139712558514176 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0811 18:59:05.196783 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:59:09.198721 139712558514176 xds_k8s_testcase.py:558] [psm-grpc-client-6fdbbf769b-6r2ck] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5878cccbb8-2q88c: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5878cccbb8-2q88c: 100

I0811 18:59:09.199013 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 18:59:09.232682 139712558514176 grpc.py:74] [psm-grpc-client-6fdbbf769b-6r2ck:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 18:59:13.238672 139712558514176 xds_k8s_testcase.py:558] [psm-grpc-client-6fdbbf769b-6r2ck] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5878cccbb8-2q88c: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5878cccbb8-2q88c: 100

I0811 18:59:13.238944 139712558514176 xds_k8s_testcase.py:524] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1691780331256735738
I0811 18:59:13.239266 139712558514176 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0811 18:59:13.239443 139712558514176 xds_k8s_testcase.py:640] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0811 18:59:13.420074 139712558514176 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1854-1msg0"
I0811 18:59:13.560843 139712558514176 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-20230811-1854-1msg0" not deleted since it does not exist
I0811 18:59:13.561091 139712558514176 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1854-1msg0"
I0811 18:59:14.169446 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780353564-602aa4f031eb3-46badf31-745fb4f8
I0811 18:59:45.125954 139712558514176 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1854-1msg0"
I0811 18:59:45.610921 139712558514176 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1854-1msg0" not deleted since it does not exist
I0811 18:59:45.611131 139712558514176 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1854-1msg0"
I0811 18:59:46.124894 139712558514176 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-20230811-1854-1msg0" not deleted since it does not exist
I0811 18:59:46.125159 139712558514176 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1854-1msg0"
I0811 18:59:46.903103 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780386127-602aa50f4003d-b9b39600-4afbaf4b
I0811 18:59:49.347963 139712558514176 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1854-1msg0"
I0811 18:59:49.778047 139712558514176 compute.py:561] Resource urlMap "psm-interop-url-map-20230811-1854-1msg0" not deleted since it does not exist
I0811 18:59:49.778267 139712558514176 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1854-1msg0"
I0811 18:59:50.327462 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780389783-602aa512bc7f9-96dc9c4c-651a9e84
I0811 18:59:52.804808 139712558514176 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1854-1msg0"
I0811 18:59:53.727601 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780392809-602aa5159f58a-17d95bf5-4028e91f
I0811 19:00:22.333481 139712558514176 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1854-1msg0"
I0811 19:00:22.784754 139712558514176 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1854-1msg0" not deleted since it does not exist
I0811 19:00:22.785069 139712558514176 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1854-1msg0"
I0811 19:00:23.245000 139712558514176 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1854-1msg0" not deleted since it does not exist
I0811 19:00:23.245241 139712558514176 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1854-1msg0"
I0811 19:00:23.806940 139712558514176 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780423328-602aa532ba1be-96280e75-ed7dbc13
I0811 19:00:28.365806 139712558514176 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:00:38.550535 139712558514176 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1854-1msg0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:00:39.438702 139712558514176 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:00:39.561972 139712558514176 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1854-1msg0
I0811 19:01:09.744380 139712558514176 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:01:19.920050 139712558514176 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:01:20.038514 139712558514176 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1854-1msg0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:01:20.723596 139712558514176 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:01:20.845082 139712558514176 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1854-1msg0
I0811 19:02:51.212564 139712558514176 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:02:51.212981 139712558514176 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-agmgx7a6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1854-1msg0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-agmgx7a6%22;timeRange=2023-08-11T18:57:14.741602Z%2F2023-08-11T19:01:09.741725Z?project=grpc-testing
I0811 19:02:51.213156 139712558514176 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-ggqbwmxx:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1854-1msg0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-ggqbwmxx%22;timeRange=2023-08-11T18:55:42.173037Z%2F2023-08-11T19:02:51.212408Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
----------------------------------------------------------------------
Ran 1 test in 481.461s

OK
+ for test in "${test_suites[@]}"
+ run_test change_backend_service_test
+ local test_name=change_backend_service_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test'
+ set -x
+ python -m tests.change_backend_service_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:02:52.186349 140418265182208 xds_k8s_testcase.py:130] ----- Testing ChangeBackendServiceTest -----
I0811 19:02:52.186485 140418265182208 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:02:52.186708 140418265182208 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:02:52.202294 140418265182208 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:02:52.217116 140418265182208 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0811 19:02:52.221418 140418265182208 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1902-bsbry
I0811 19:02:52.570673 140418265182208 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
I0811 19:02:52.877914 140418265182208 xds_k8s_testcase.py:625] Found unused xds port: 11407
I0811 19:02:52.880177 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0811 19:02:52.880379 140418265182208 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1902-bsbry"
I0811 19:02:52.883401 140418265182208 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1902-bsbry
type: GRPC
...

I0811 19:02:53.516752 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780572952-602aa5c16b967-6661a2e2-476ce8c0
I0811 19:02:55.882738 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0811 19:02:55.882952 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0811 19:02:55.883044 140418265182208 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1902-bsbry"
I0811 19:02:55.890175 140418265182208 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1902-bsbry
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1902-bsbry
protocol: GRPC
...

I0811 19:02:56.513833 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780575892-602aa5c4394a4-393a9055-42b26f40
I0811 19:03:12.184661 140418265182208 traffic_director.py:305] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20230811-1902-bsbry"
I0811 19:03:12.189222 140418265182208 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1902-bsbry
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20230811-1902-bsbry
protocol: GRPC
...

I0811 19:03:13.076797 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780592196-602aa5d3c5ada-88a5492f-6762f887
I0811 19:03:35.149580 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0811 19:03:35.149819 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0811 19:03:35.149913 140418265182208 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1902-bsbry": psm-grpc-server:11407 -> psm-interop-backend-service-20230811-1902-bsbry
I0811 19:03:35.160785 140418265182208 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1902-bsbry
hostRules:
- hosts:
- psm-grpc-server:11407
pathMatcher: psm-interop-path-matcher-20230811-1902-bsbry
name: psm-interop-url-map-20230811-1902-bsbry
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1902-bsbry
name: psm-interop-path-matcher-20230811-1902-bsbry
...

I0811 19:03:35.730483 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780615162-602aa5e9acc35-d86ea9c4-2253ec42
I0811 19:03:38.117633 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0811 19:03:38.117835 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0811 19:03:38.117927 140418265182208 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1902-bsbry proxy "GRPC" to URL map psm-interop-url-map-20230811-1902-bsbry
I0811 19:03:38.119957 140418265182208 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1902-bsbry
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1902-bsbry
validate_for_proxyless: true
...

I0811 19:03:38.990721 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780618122-602aa5ec7f610-5889bacd-2e28287e
I0811 19:03:41.484099 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0811 19:03:41.484328 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0811 19:03:41.484439 140418265182208 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1902-bsbry" in network "default-vpc": 0.0.0.0:11407 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1902-bsbry
I0811 19:03:41.486986 140418265182208 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1902-bsbry
network: global/networks/default-vpc
portRange: 11407
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1902-bsbry
...

I0811 19:03:42.482049 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780621489-602aa5efb550d-cec6ebd6-9ad2031f
I0811 19:03:55.920431 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0811 19:03:55.920645 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0811 19:03:55.920743 140418265182208 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1902-bsbry: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:03:55.920995 140418265182208 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1902-bsbry%22;timeRange=2023-08-11T19:03:55.920818Z%2F2023-08-11T19:33:55.920818Z?project=grpc-testing
I0811 19:03:55.925846 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1902-bsbry
labels:
name: psm-interop-server-20230811-1902-bsbry
owner: xds-k8s-interop-test
...

I0811 19:03:56.058465 140418265182208 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1902-bsbry created
I0811 19:03:56.061370 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1902-bsbry
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1902-bsbry-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:03:56.134245 140418265182208 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:03:56.134478 140418265182208 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:03:56.240363 140418265182208 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1902-bsbry-psm-grpc-server in zones=['us-central1-a']
I0811 19:03:56.240561 140418265182208 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1902-bsbry/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:03:56.937201 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1902-bsbry
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:03:56.998257 140418265182208 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:03:57.006252 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1902-bsbry
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-gxnoe7ij
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-gxnoe7ij
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-gxnoe7ij
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1902-bsbry-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:03:57.086010 140418265182208 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:03:57.086226 140418265182208 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 19:03:57.190697 140418265182208 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6447747964-c7rnp']
I0811 19:03:57.190956 140418265182208 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-6447747964-c7rnp to start
I0811 19:04:00.425593 140418265182208 k8s_base_runner.py:554] Pod psm-grpc-server-6447747964-c7rnp ready, IP: 10.120.11.233
I0811 19:04:00.425853 140418265182208 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-6447747964-c7rnp to change_backend_service_test/test_app_logs/psm-interop-server-20230811-1902-bsbry_psm-grpc-server-6447747964-c7rnp.log
I0811 19:04:00.426332 140418174412352 k8s_log_collector.py:70] Starting log collection thread 140418174412352 for psm-grpc-server-6447747964-c7rnp
I0811 19:04:00.426509 140418265182208 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 19:04:05.797713 140418265182208 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 19:04:05.798197 140418265182208 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230811-1902-bsbry: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:04:05.798440 140418265182208 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1902-bsbry%22;timeRange=2023-08-11T19:04:05.798323Z%2F2023-08-11T19:34:05.798323Z?project=grpc-testing
I0811 19:04:05.858400 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1902-bsbry
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:04:05.902510 140418265182208 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0811 19:04:05.902723 140418265182208 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0811 19:04:05.998827 140418265182208 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt in zones=['us-central1-a']
I0811 19:04:05.999031 140418265182208 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1902-bsbry/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:04:06.691591 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1902-bsbry
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:04:06.731910 140418265182208 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0811 19:04:06.740028 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1902-bsbry
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-60gl5iis
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-60gl5iis
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-60gl5iis
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:04:06.791952 140418265182208 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0811 19:04:06.792178 140418265182208 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0811 19:04:06.877450 140418265182208 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:04:06.877702 140418265182208 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-c55cd5cc6-hf27z to start
I0811 19:04:10.116706 140418265182208 k8s_base_runner.py:554] Pod psm-grpc-server-alt-c55cd5cc6-hf27z ready, IP: 10.120.17.229
I0811 19:04:10.116980 140418265182208 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-c55cd5cc6-hf27z to change_backend_service_test/test_app_logs/psm-interop-server-20230811-1902-bsbry_psm-grpc-server-alt-c55cd5cc6-hf27z.log
I0811 19:04:10.117403 140418166019648 k8s_log_collector.py:70] Starting log collection thread 140418166019648 for psm-grpc-server-alt-c55cd5cc6-hf27z
I0811 19:04:10.117549 140418265182208 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0811 19:04:16.534129 140418265182208 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0811 19:04:16.534537 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0811 19:04:16.534647 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0811 19:04:16.571454 140418265182208 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:04:16.845439 140418265182208 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1902-bsbry-psm-grpc-server" in zone us-central1-a
I0811 19:04:16.845664 140418265182208 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1902-bsbry: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1902-bsbry-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1902-bsbry-psm-grpc-server', zone='us-central1-a')}
I0811 19:04:16.849582 140418265182208 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1902-bsbry-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:04:17.948342 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780656851-602aa6116ea94-c5b05ba8-02a8c3b6
I0811 19:04:52.770706 140418265182208 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1902-bsbry to report all backends healthy: ['psm-interop-server-20230811-1902-bsbry-psm-grpc-server']
I0811 19:04:57.239871 140418265182208 compute.py:487] Backend psm-interop-server-20230811-1902-bsbry-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:05:01.338630 140418265182208 traffic_director.py:333] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:05:01.632277 140418265182208 traffic_director.py:336] Loaded NEG "psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt" in zone us-central1-a
I0811 19:05:01.632612 140418265182208 traffic_director.py:343] Adding backends to Backend Service psm-interop-backend-service-alt-20230811-1902-bsbry: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt', zone='us-central1-a')}
I0811 19:05:01.636687 140418265182208 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1902-bsbry-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0811 19:05:02.648271 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780701638-602aa63c25031-c2d5e157-9a6af10a
I0811 19:05:42.364416 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0811 19:05:42.364664 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0811 19:05:42.364770 140418265182208 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1902-bsbry: server_target=xds:///psm-grpc-server:11407 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 19:05:42.365075 140418265182208 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1902-bsbry%22;timeRange=2023-08-11T19:05:42.364846Z%2F2023-08-11T19:35:42.364846Z?project=grpc-testing
I0811 19:05:42.366986 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1902-bsbry
labels:
name: psm-interop-client-20230811-1902-bsbry
owner: xds-k8s-interop-test
...

I0811 19:05:42.427888 140418265182208 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1902-bsbry created
I0811 19:05:42.428129 140418265182208 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1902-bsbry/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:05:43.331123 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1902-bsbry
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:05:43.389933 140418265182208 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:05:43.398083 140418265182208 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1902-bsbry
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zx2vuio1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zx2vuio1
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zx2vuio1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:11407"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:05:43.466417 140418265182208 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:05:43.466664 140418265182208 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:05:43.569481 140418265182208 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-69b747bb4d-hw9lx']
I0811 19:05:43.569760 140418265182208 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-69b747bb4d-hw9lx to start
I0811 19:05:46.799190 140418265182208 k8s_base_runner.py:554] Pod psm-grpc-client-69b747bb4d-hw9lx ready, IP: 10.120.13.240
I0811 19:05:46.799488 140418265182208 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-69b747bb4d-hw9lx to change_backend_service_test/test_app_logs/psm-interop-client-20230811-1902-bsbry_psm-grpc-client-69b747bb4d-hw9lx.log
I0811 19:05:46.799911 140418157626944 k8s_log_collector.py:70] Starting log collection thread 140418157626944 for psm-grpc-client-69b747bb4d-hw9lx
I0811 19:05:46.800051 140418265182208 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:05:47.934373 140418265182208 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:05:47.934922 140418265182208 client_app.py:200] [psm-grpc-client-69b747bb4d-hw9lx] Waiting to report a READY channel to xds:///psm-grpc-server:11407
I0811 19:05:48.035740 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:05:58.080616 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:06:08.124651 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:06:18.169507 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:06:28.214879 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:06:44.264553 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:07:09.323966 140418265182208 client_app.py:236] [psm-grpc-client-69b747bb4d-hw9lx] Server channel:
I0811 19:07:09.357005 140418265182208 client_app.py:249] [psm-grpc-client-69b747bb4d-hw9lx] Found subchannel in state READY:
I0811 19:07:09.357309 140418265182208 client_app.py:211] [psm-grpc-client-69b747bb4d-hw9lx] Channel to xds:///psm-grpc-server:11407 transitioned to state READY:
I0811 19:07:09.357488 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0811 19:07:09.357578 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0811 19:07:09.357766 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0811 19:07:09.392707 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0811 19:07:09.392964 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0811 19:07:09.393179 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:13.426834 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:13.427087 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0811 19:07:13.427178 140418265182208 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0811 19:07:13.427261 140418265182208 traffic_director.py:495] Patching URL map "psm-interop-url-map-20230811-1902-bsbry": psm-grpc-server:11407 -> psm-interop-backend-service-alt-20230811-1902-bsbry
I0811 19:07:13.432351 140418265182208 compute.py:537] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20230811-1902-bsbry
hostRules:
- hosts:
- psm-grpc-server:11407
pathMatcher: psm-interop-path-matcher-20230811-1902-bsbry
name: psm-interop-url-map-20230811-1902-bsbry
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20230811-1902-bsbry
name: psm-interop-path-matcher-20230811-1902-bsbry
...

I0811 19:07:14.375329 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780833434-602aa6b9d5c9a-57233d80-0df47604
I0811 19:07:23.095963 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:23.096270 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:27.107702 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:27.108364 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-c55cd5cc6-hf27z' not found in {'psm-grpc-server-6447747964-c7rnp': 100} : Server psm-grpc-server-alt-c55cd5cc6-hf27z did not receive RPCs.
I0811 19:07:28.109713 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:28.110072 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:32.106965 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:32.107344 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-c55cd5cc6-hf27z' not found in {'psm-grpc-server-6447747964-c7rnp': 100} : Server psm-grpc-server-alt-c55cd5cc6-hf27z did not receive RPCs.
I0811 19:07:33.108703 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:33.109133 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:37.107258 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:37.107647 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-c55cd5cc6-hf27z' not found in {'psm-grpc-server-6447747964-c7rnp': 100} : Server psm-grpc-server-alt-c55cd5cc6-hf27z did not receive RPCs.
I0811 19:07:38.109001 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:38.109484 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:42.106572 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:42.106992 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-c55cd5cc6-hf27z' not found in {'psm-grpc-server-6447747964-c7rnp': 100} : Server psm-grpc-server-alt-c55cd5cc6-hf27z did not receive RPCs.
I0811 19:07:43.108362 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:43.108732 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:47.107097 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 100

I0811 19:07:47.107515 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-c55cd5cc6-hf27z' not found in {'psm-grpc-server-6447747964-c7rnp': 100} : Server psm-grpc-server-alt-c55cd5cc6-hf27z did not receive RPCs.
I0811 19:07:48.108838 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:48.109225 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:53.328370 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6447747964-c7rnp: 64
psm-grpc-server-alt-c55cd5cc6-hf27z: 36
rpcs_by_method:
UnaryCall:
psm-grpc-server-6447747964-c7rnp: 64
psm-grpc-server-alt-c55cd5cc6-hf27z: 36

I0811 19:07:53.328862 140418265182208 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6447747964-c7rnp' not found in ['psm-grpc-server-alt-c55cd5cc6-hf27z'] : Unexpected server psm-grpc-server-6447747964-c7rnp received RPCs.
I0811 19:07:54.330245 140418265182208 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-c55cd5cc6-hf27z']
I0811 19:07:54.330626 140418265182208 grpc.py:74] [psm-grpc-client-69b747bb4d-hw9lx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:07:58.347063 140418265182208 xds_k8s_testcase.py:558] [psm-grpc-client-69b747bb4d-hw9lx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-c55cd5cc6-hf27z: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-c55cd5cc6-hf27z: 100

I0811 19:07:58.347608 140418265182208 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0811 19:07:58.347846 140418265182208 xds_k8s_testcase.py:640] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0811 19:07:58.575469 140418265182208 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1902-bsbry"
I0811 19:07:59.431602 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780878580-602aa6e4e39e9-622cf431-55d916dd
I0811 19:08:12.834576 140418265182208 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1902-bsbry"
I0811 19:08:12.953308 140418265182208 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1902-bsbry" not deleted since it does not exist
I0811 19:08:12.953531 140418265182208 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1902-bsbry"
I0811 19:08:13.405040 140418265182208 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1902-bsbry" not deleted since it does not exist
I0811 19:08:13.405256 140418265182208 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1902-bsbry"
I0811 19:08:13.813570 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780893408-602aa6f307c38-3a0e32f9-02a52421
I0811 19:08:16.153517 140418265182208 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1902-bsbry"
I0811 19:08:16.716664 140418265182208 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1902-bsbry" not deleted since it does not exist
I0811 19:08:16.716919 140418265182208 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1902-bsbry"
I0811 19:08:17.446471 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780896723-602aa6f631171-ce8d836b-445b3cc6
I0811 19:08:19.790731 140418265182208 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1902-bsbry"
I0811 19:08:20.297556 140418265182208 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1902-bsbry" not deleted since it does not exist
I0811 19:08:20.297764 140418265182208 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1902-bsbry"
I0811 19:08:20.901113 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780900302-602aa6f99af29-028ada33-bdada074
I0811 19:08:47.213609 140418265182208 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1902-bsbry"
I0811 19:08:48.011206 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780927218-602aa713463d3-11857b16-8cd4f0a3
I0811 19:09:12.151698 140418265182208 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1902-bsbry"
I0811 19:09:12.616933 140418265182208 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1902-bsbry" not deleted since it does not exist
I0811 19:09:12.617196 140418265182208 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1902-bsbry"
I0811 19:09:13.199749 140418265182208 compute.py:589] Waiting 600 sec for compute operation id: operation-1691780952696-602aa72b926a6-078015c7-c71c84f4
I0811 19:09:17.739995 140418265182208 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:09:27.917488 140418265182208 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1902-bsbry/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:09:28.716192 140418265182208 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:09:28.847792 140418265182208 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1902-bsbry
I0811 19:09:59.070282 140418265182208 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:10:09.242096 140418265182208 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:10:09.361853 140418265182208 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1902-bsbry/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:10:09.989476 140418265182208 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:10:10.101402 140418265182208 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1902-bsbry
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/exceptions.py:91: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
self.headers = http_resp.getheaders()
I0811 19:11:40.419345 140418265182208 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
I0811 19:11:40.477055 140418265182208 k8s_base_runner.py:466] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0811 19:11:40.477364 140418265182208 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
I0811 19:11:40.537144 140418265182208 k8s_base_runner.py:478] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0811 19:11:40.537404 140418265182208 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1902-bsbry/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:11:41.195141 140418265182208 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
I0811 19:11:41.251797 140418265182208 k8s_base_runner.py:491] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0811 19:11:41.252057 140418265182208 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1902-bsbry
I0811 19:11:41.287241 140418265182208 k8s_base_runner.py:503] Namespace psm-interop-server-20230811-1902-bsbry deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230811-1902-bsbry" not found
I0811 19:11:41.287663 140418265182208 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:11:41.287921 140418265182208 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-zx2vuio1:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1902-bsbry%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-zx2vuio1%22;timeRange=2023-08-11T19:05:42.364838Z%2F2023-08-11T19:09:59.066590Z?project=grpc-testing
I0811 19:11:41.288100 140418265182208 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-gxnoe7ij:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1902-bsbry%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-gxnoe7ij%22;timeRange=2023-08-11T19:03:55.920811Z%2F2023-08-11T19:11:40.419037Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
----------------------------------------------------------------------
Ran 1 test in 529.104s

OK
+ for test in "${test_suites[@]}"
+ run_test custom_lb_test
+ local test_name=custom_lb_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test'
+ set -x
+ python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:11:42.294852 140483974373376 xds_k8s_testcase.py:130] ----- Testing CustomLbTest -----
I0811 19:11:42.295012 140483974373376 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:11:42.295249 140483974373376 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:11:42.311433 140483974373376 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:11:42.327183 140483974373376 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0811 19:11:42.331520 140483974373376 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1911-52uuo
I0811 19:11:42.723857 140483974373376 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
I0811 19:11:43.108357 140483974373376 xds_k8s_testcase.py:625] Found unused xds port: 19030
I0811 19:11:43.108707 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0811 19:11:43.108784 140483974373376 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1911-52uuo"
I0811 19:11:43.111921 140483974373376 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1911-52uuo
type: GRPC
...

I0811 19:11:43.725029 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781103173-602aa7bb13dc4-c6c8fe31-a4266ace
I0811 19:11:46.113617 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0811 19:11:46.113850 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0811 19:11:46.113952 140483974373376 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1911-52uuo"
I0811 19:11:46.121968 140483974373376 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1911-52uuo
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20230811-1911-52uuo
protocol: GRPC
...

I0811 19:11:47.129500 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781106124-602aa7bde4583-13252067-9505f82a
I0811 19:12:09.057210 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0811 19:12:09.057444 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0811 19:12:09.057536 140483974373376 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1911-52uuo": psm-grpc-server:19030 -> psm-interop-backend-service-20230811-1911-52uuo
I0811 19:12:09.068752 140483974373376 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1911-52uuo
hostRules:
- hosts:
- psm-grpc-server:19030
pathMatcher: psm-interop-path-matcher-20230811-1911-52uuo
name: psm-interop-url-map-20230811-1911-52uuo
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1911-52uuo
name: psm-interop-path-matcher-20230811-1911-52uuo
...

I0811 19:12:09.702217 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781129070-602aa7d3c697f-5f3d85bc-5ca2cc0b
I0811 19:12:12.005259 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0811 19:12:12.005487 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0811 19:12:12.005576 140483974373376 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1911-52uuo proxy "GRPC" to URL map psm-interop-url-map-20230811-1911-52uuo
I0811 19:12:12.007447 140483974373376 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1911-52uuo
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1911-52uuo
validate_for_proxyless: true
...

I0811 19:12:12.805440 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781132009-602aa7d693f82-6d704d8e-e6a03e1c
I0811 19:12:15.296575 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0811 19:12:15.296797 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0811 19:12:15.296897 140483974373376 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1911-52uuo" in network "default-vpc": 0.0.0.0:19030 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1911-52uuo
I0811 19:12:15.299446 140483974373376 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1911-52uuo
network: global/networks/default-vpc
portRange: 19030
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1911-52uuo
...

I0811 19:12:16.326952 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781135301-602aa7d9b7b56-c9c6b480-391b024c
I0811 19:13:06.666838 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0811 19:13:06.667081 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0811 19:13:06.667181 140483974373376 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1911-52uuo: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:13:06.667417 140483974373376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1911-52uuo%22;timeRange=2023-08-11T19:13:06.667257Z%2F2023-08-11T19:43:06.667257Z?project=grpc-testing
I0811 19:13:06.672325 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1911-52uuo
labels:
name: psm-interop-server-20230811-1911-52uuo
owner: xds-k8s-interop-test
...

I0811 19:13:06.821296 140483974373376 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1911-52uuo created
I0811 19:13:06.824223 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1911-52uuo
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1911-52uuo-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:13:06.891936 140483974373376 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:13:06.892187 140483974373376 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:13:06.988896 140483974373376 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1911-52uuo-psm-grpc-server in zones=['us-central1-a']
I0811 19:13:06.989138 140483974373376 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1911-52uuo/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:13:07.920174 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1911-52uuo
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:13:07.988765 140483974373376 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:13:07.996760 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1911-52uuo
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-dvmowijt
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-dvmowijt
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-dvmowijt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1911-52uuo-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:13:08.067805 140483974373376 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:13:08.068036 140483974373376 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 19:13:08.171856 140483974373376 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-7d859d4949-kkgsb']
I0811 19:13:08.172115 140483974373376 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-7d859d4949-kkgsb to start
I0811 19:13:11.411871 140483974373376 k8s_base_runner.py:554] Pod psm-grpc-server-7d859d4949-kkgsb ready, IP: 10.120.11.234
I0811 19:13:11.412121 140483974373376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-7d859d4949-kkgsb to custom_lb_test/test_app_logs/psm-interop-server-20230811-1911-52uuo_psm-grpc-server-7d859d4949-kkgsb.log
I0811 19:13:11.412575 140483882952256 k8s_log_collector.py:70] Starting log collection thread 140483882952256 for psm-grpc-server-7d859d4949-kkgsb
I0811 19:13:11.412755 140483974373376 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 19:13:18.861286 140483974373376 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 19:13:18.861696 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0811 19:13:18.861803 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0811 19:13:18.897892 140483974373376 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:13:19.188081 140483974373376 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1911-52uuo-psm-grpc-server" in zone us-central1-a
I0811 19:13:19.188303 140483974373376 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1911-52uuo: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1911-52uuo-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1911-52uuo-psm-grpc-server', zone='us-central1-a')}
I0811 19:13:19.192268 140483974373376 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1911-52uuo-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:13:20.058058 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781199194-602aa816a68ea-9f04bff6-2b94c3c2
I0811 19:14:05.610496 140483974373376 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1911-52uuo to report all backends healthy: ['psm-interop-server-20230811-1911-52uuo-psm-grpc-server']
I0811 19:14:14.269388 140483974373376 compute.py:487] Backend psm-interop-server-20230811-1911-52uuo-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:14:18.274032 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0811 19:14:18.274265 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0811 19:14:18.274400 140483974373376 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1911-52uuo: server_target=xds:///psm-grpc-server:19030 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 19:14:18.274614 140483974373376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1911-52uuo%22;timeRange=2023-08-11T19:14:18.274485Z%2F2023-08-11T19:44:18.274485Z?project=grpc-testing
I0811 19:14:18.276457 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1911-52uuo
labels:
name: psm-interop-client-20230811-1911-52uuo
owner: xds-k8s-interop-test
...

I0811 19:14:18.317579 140483974373376 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1911-52uuo created
I0811 19:14:18.317798 140483974373376 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1911-52uuo/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:14:19.053541 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1911-52uuo
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:14:19.113413 140483974373376 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:14:19.121078 140483974373376 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1911-52uuo
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ekfoffx9
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ekfoffx9
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ekfoffx9
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:19030"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:14:19.195478 140483974373376 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:14:19.195722 140483974373376 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:14:19.304071 140483974373376 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5d57d5cf98-bnxs4']
I0811 19:14:19.304369 140483974373376 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-5d57d5cf98-bnxs4 to start
I0811 19:14:22.539093 140483974373376 k8s_base_runner.py:554] Pod psm-grpc-client-5d57d5cf98-bnxs4 ready, IP: 10.120.13.241
I0811 19:14:22.539343 140483974373376 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-5d57d5cf98-bnxs4 to custom_lb_test/test_app_logs/psm-interop-client-20230811-1911-52uuo_psm-grpc-client-5d57d5cf98-bnxs4.log
I0811 19:14:22.539769 140483874559552 k8s_log_collector.py:70] Starting log collection thread 140483874559552 for psm-grpc-client-5d57d5cf98-bnxs4
I0811 19:14:22.540012 140483974373376 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:14:23.751563 140483974373376 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:14:23.752127 140483974373376 client_app.py:200] [psm-grpc-client-5d57d5cf98-bnxs4] Waiting to report a READY channel to xds:///psm-grpc-server:19030
I0811 19:14:23.854460 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:14:33.900352 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:14:43.946485 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:14:53.991641 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:15:04.037077 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:15:20.086903 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:15:45.147595 140483974373376 client_app.py:236] [psm-grpc-client-5d57d5cf98-bnxs4] Server channel:
I0811 19:15:45.180434 140483974373376 client_app.py:249] [psm-grpc-client-5d57d5cf98-bnxs4] Found subchannel in state READY:
I0811 19:15:45.180734 140483974373376 client_app.py:211] [psm-grpc-client-5d57d5cf98-bnxs4] Channel to xds:///psm-grpc-server:19030 transitioned to state READY:
I0811 19:15:45.180875 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0811 19:15:45.180974 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0811 19:15:45.181169 140483974373376 grpc.py:74] [psm-grpc-client-5d57d5cf98-bnxs4:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 19:15:45.216474 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0811 19:15:45.216694 140483974373376 xds_k8s_testcase.py:239] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0811 19:15:45.216888 140483974373376 grpc.py:74] [psm-grpc-client-5d57d5cf98-bnxs4:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0811 19:15:55.261437 140483974373376 grpc.py:74] [psm-grpc-client-5d57d5cf98-bnxs4:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0811 19:15:55.297080 140483974373376 xds_k8s_testcase.py:361] [psm-grpc-client-5d57d5cf98-bnxs4] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 251
result:
(15, DATA_LOSS): 251

I0811 19:15:55.297785 140483974373376 xds_k8s_testcase.py:244] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0811 19:15:55.297961 140483974373376 xds_k8s_testcase.py:640] ----- TestMethod __main__.CustomLbTest.test_custom_lb_config teardown -----
I0811 19:15:55.526175 140483974373376 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1911-52uuo"
I0811 19:15:56.278728 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781355530-602aa8abbe798-ea8f7113-6071ad2f
I0811 19:16:26.864143 140483974373376 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1911-52uuo"
I0811 19:16:27.038835 140483974373376 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:16:27.039078 140483974373376 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1911-52uuo"
I0811 19:16:27.424568 140483974373376 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:16:27.424789 140483974373376 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1911-52uuo"
I0811 19:16:28.041500 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781387428-602aa8ca2a3b8-e6a6fb16-4bc9f669
I0811 19:16:30.253038 140483974373376 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1911-52uuo"
I0811 19:16:30.732833 140483974373376 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:16:30.733038 140483974373376 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1911-52uuo"
I0811 19:16:31.403646 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781390738-602aa8cd52455-cc8b5fa5-de891bcf
I0811 19:16:33.767953 140483974373376 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1911-52uuo"
I0811 19:16:34.120938 140483974373376 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:16:34.121195 140483974373376 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1911-52uuo"
I0811 19:16:34.725668 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781394127-602aa8d08d987-ffd3dce6-af5525ca
I0811 19:17:01.421077 140483974373376 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1911-52uuo"
I0811 19:17:01.904188 140483974373376 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:17:01.904458 140483974373376 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1911-52uuo"
I0811 19:17:02.386936 140483974373376 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1911-52uuo" not deleted since it does not exist
I0811 19:17:02.387160 140483974373376 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1911-52uuo"
I0811 19:17:02.916827 140483974373376 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781422482-602aa8eb98337-98eb0f90-f5f172e6
I0811 19:17:07.430865 140483974373376 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:17:17.616799 140483974373376 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1911-52uuo/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:17:18.408534 140483974373376 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:17:18.529838 140483974373376 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1911-52uuo
I0811 19:17:48.717159 140483974373376 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:17:58.941480 140483974373376 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:17:59.062395 140483974373376 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1911-52uuo/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:17:59.659685 140483974373376 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:17:59.774226 140483974373376 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1911-52uuo
I0811 19:20:00.175061 140483974373376 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:20:00.175482 140483974373376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-ekfoffx9:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1911-52uuo%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-ekfoffx9%22;timeRange=2023-08-11T19:14:18.274478Z%2F2023-08-11T19:17:48.711768Z?project=grpc-testing
I0811 19:20:00.175657 140483974373376 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-dvmowijt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1911-52uuo%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-dvmowijt%22;timeRange=2023-08-11T19:13:06.667248Z%2F2023-08-11T19:20:00.174899Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
----------------------------------------------------------------------
Ran 1 test in 497.884s

OK
+ for test in "${test_suites[@]}"
+ run_test failover_test
+ local test_name=failover_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test'
+ set -x
+ python -m tests.failover_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:20:01.116274 139809716162560 xds_k8s_testcase.py:130] ----- Testing FailoverTest -----
I0811 19:20:01.116415 139809716162560 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:20:01.116638 139809716162560 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:20:01.132711 139809716162560 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:20:01.148027 139809716162560 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0811 19:20:01.152437 139809716162560 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1920-2z4i0
I0811 19:20:01.536471 139809716162560 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
I0811 19:20:01.824470 139809716162560 xds_k8s_testcase.py:625] Found unused xds port: 9055
I0811 19:20:01.826761 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0811 19:20:01.826946 139809716162560 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1920-2z4i0"
I0811 19:20:01.830114 139809716162560 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1920-2z4i0
type: GRPC
...

I0811 19:20:02.491830 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781601885-602aa996afd6f-446123ee-84c702b5
I0811 19:20:04.788927 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0811 19:20:04.789164 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0811 19:20:04.789257 139809716162560 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1920-2z4i0"
I0811 19:20:04.796468 139809716162560 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1920-2z4i0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1920-2z4i0
protocol: GRPC
...

I0811 19:20:05.443162 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781604799-602aa9997739d-1cc6d262-59aca6cc
I0811 19:20:25.271373 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0811 19:20:25.271615 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0811 19:20:25.271708 139809716162560 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1920-2z4i0": psm-grpc-server:9055 -> psm-interop-backend-service-20230811-1920-2z4i0
I0811 19:20:25.282929 139809716162560 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1920-2z4i0
hostRules:
- hosts:
- psm-grpc-server:9055
pathMatcher: psm-interop-path-matcher-20230811-1920-2z4i0
name: psm-interop-url-map-20230811-1920-2z4i0
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1920-2z4i0
name: psm-interop-path-matcher-20230811-1920-2z4i0
...

I0811 19:20:26.050694 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781625285-602aa9ad00aa0-6bdf591e-3836ee35
I0811 19:20:28.471089 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0811 19:20:28.471332 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0811 19:20:28.471441 139809716162560 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1920-2z4i0 proxy "GRPC" to URL map psm-interop-url-map-20230811-1920-2z4i0
I0811 19:20:28.473362 139809716162560 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1920-2z4i0
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1920-2z4i0
validate_for_proxyless: true
...

I0811 19:20:29.196569 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781628475-602aa9b00b8fc-7657f768-3b8eca5d
I0811 19:20:31.503699 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0811 19:20:31.503930 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0811 19:20:31.504037 139809716162560 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1920-2z4i0" in network "default-vpc": 0.0.0.0:9055 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1920-2z4i0
I0811 19:20:31.506754 139809716162560 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1920-2z4i0
network: global/networks/default-vpc
portRange: 9055
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1920-2z4i0
...

I0811 19:20:32.504880 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781631509-602aa9b2f028e-97974d34-6c78aeb0
I0811 19:21:03.013604 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0811 19:21:03.013848 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0811 19:21:03.013955 139809716162560 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1920-2z4i0: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0811 19:21:03.014217 139809716162560 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1920-2z4i0%22;timeRange=2023-08-11T19:21:03.014050Z%2F2023-08-11T19:51:03.014050Z?project=grpc-testing
I0811 19:21:03.019116 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1920-2z4i0
labels:
name: psm-interop-server-20230811-1920-2z4i0
owner: xds-k8s-interop-test
...

I0811 19:21:03.146293 139809716162560 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1920-2z4i0 created
I0811 19:21:03.149345 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1920-2z4i0-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:21:03.223721 139809716162560 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:21:03.223944 139809716162560 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:21:03.315327 139809716162560 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1920-2z4i0-psm-grpc-server in zones=['us-central1-a']
I0811 19:21:03.315561 139809716162560 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1920-2z4i0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:21:04.164259 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:21:04.220695 139809716162560 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:21:04.228939 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-97w3yjq6
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-97w3yjq6
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-97w3yjq6
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1920-2z4i0-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:21:04.304523 139809716162560 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:21:04.304764 139809716162560 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0811 19:21:04.469717 139809716162560 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:21:04.470099 139809716162560 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5bbc4d5d48-fqzxd to start
I0811 19:21:07.700723 139809716162560 k8s_base_runner.py:554] Pod psm-grpc-server-5bbc4d5d48-fqzxd ready, IP: 10.120.14.167
I0811 19:21:07.701037 139809716162560 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5bbc4d5d48-fqzxd to failover_test/test_app_logs/psm-interop-server-20230811-1920-2z4i0_psm-grpc-server-5bbc4d5d48-fqzxd.log
I0811 19:21:07.701531 139809625990720 k8s_log_collector.py:70] Starting log collection thread 139809625990720 for psm-grpc-server-5bbc4d5d48-fqzxd
I0811 19:21:07.701678 139809716162560 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5bbc4d5d48-qkctk to start
I0811 19:21:08.855064 139809716162560 k8s_base_runner.py:554] Pod psm-grpc-server-5bbc4d5d48-qkctk ready, IP: 10.120.13.242
I0811 19:21:08.855328 139809716162560 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5bbc4d5d48-qkctk to failover_test/test_app_logs/psm-interop-server-20230811-1920-2z4i0_psm-grpc-server-5bbc4d5d48-qkctk.log
I0811 19:21:08.855756 139809617598016 k8s_log_collector.py:70] Starting log collection thread 139809617598016 for psm-grpc-server-5bbc4d5d48-qkctk
I0811 19:21:08.855894 139809716162560 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5bbc4d5d48-ql456 to start
I0811 19:21:08.962329 139809716162560 k8s_base_runner.py:554] Pod psm-grpc-server-5bbc4d5d48-ql456 ready, IP: 10.120.9.226
I0811 19:21:08.962586 139809716162560 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5bbc4d5d48-ql456 to failover_test/test_app_logs/psm-interop-server-20230811-1920-2z4i0_psm-grpc-server-5bbc4d5d48-ql456.log
I0811 19:21:08.963014 139809609205312 k8s_log_collector.py:70] Starting log collection thread 139809609205312 for psm-grpc-server-5bbc4d5d48-ql456
I0811 19:21:08.963186 139809716162560 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0811 19:21:14.338724 139809716162560 k8s_base_runner.py:522] Deployment psm-grpc-server has 3 replicas available
I0811 19:21:14.339278 139809716162560 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230811-1920-2z4i0: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:21:14.339537 139809716162560 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1920-2z4i0%22;timeRange=2023-08-11T19:21:14.339398Z%2F2023-08-11T19:51:14.339398Z?project=grpc-testing
I0811 19:21:14.341373 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1920-2z4i0
labels:
name: psm-interop-server-20230811-1920-2z4i0
owner: xds-k8s-interop-test
...

I0811 19:21:14.594085 139809716162560 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1920-2z4i0 created
I0811 19:21:14.597045 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:21:14.701435 139809716162560 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0811 19:21:14.701656 139809716162560 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0811 19:21:14.861073 139809716162560 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt in zones=['us-west1-b']
I0811 19:21:14.861292 139809716162560 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1920-2z4i0/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:21:15.600437 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:21:15.692295 139809716162560 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0811 19:21:15.699686 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1920-2z4i0
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-vx5f1m76
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-vx5f1m76
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-vx5f1m76
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:21:15.807898 139809716162560 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0811 19:21:15.808134 139809716162560 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0811 19:21:15.986145 139809716162560 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:21:15.986421 139809716162560 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-76d547cff6-qtpbm to start
I0811 19:21:22.630873 139809716162560 k8s_base_runner.py:554] Pod psm-grpc-server-alt-76d547cff6-qtpbm ready, IP: 10.108.5.246
I0811 19:21:22.631139 139809716162560 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-76d547cff6-qtpbm to failover_test/test_app_logs/psm-interop-server-20230811-1920-2z4i0_psm-grpc-server-alt-76d547cff6-qtpbm.log
I0811 19:21:22.631567 139809600812608 k8s_log_collector.py:70] Starting log collection thread 139809600812608 for psm-grpc-server-alt-76d547cff6-qtpbm
I0811 19:21:22.631705 139809716162560 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0811 19:21:27.229394 139809716162560 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0811 19:21:27.229802 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0811 19:21:27.229915 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0811 19:21:27.267954 139809716162560 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:21:27.394066 139809716162560 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1920-2z4i0-psm-grpc-server" in zone us-central1-a
I0811 19:21:27.394293 139809716162560 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1920-2z4i0: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1920-2z4i0-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server', zone='us-central1-a')}
I0811 19:21:27.398222 139809716162560 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server
maxRatePerEndpoint: 100
...

I0811 19:21:28.341470 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781687400-602aa9e83d7b6-b0e4ca13-0b80ac4a
I0811 19:22:26.850018 139809716162560 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1920-2z4i0 to report all backends healthy: ['psm-interop-server-20230811-1920-2z4i0-psm-grpc-server']
I0811 19:22:35.499962 139809716162560 compute.py:487] Backend psm-interop-server-20230811-1920-2z4i0-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:22:39.575695 139809716162560 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:22:39.793114 139809716162560 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt" in zone us-west1-b
I0811 19:22:39.793344 139809716162560 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1920-2z4i0: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1920-2z4i0-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server', zone='us-central1-a')}
I0811 19:22:39.798158 139809716162560 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1920-2z4i0-psm-grpc-server
maxRatePerEndpoint: 100
...

I0811 19:22:41.489817 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781759800-602aaa2d49603-1402816a-cab77b96
I0811 19:23:40.124732 139809716162560 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1920-2z4i0 to report all backends healthy: ['psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt', 'psm-interop-server-20230811-1920-2z4i0-psm-grpc-server']
I0811 19:23:40.972817 139809716162560 compute.py:487] Backend psm-interop-server-20230811-1920-2z4i0-psm-grpc-server-alt in zone us-west1-b reported healthy
I0811 19:23:45.574081 139809716162560 compute.py:487] Backend psm-interop-server-20230811-1920-2z4i0-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:23:49.578777 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0811 19:23:49.579019 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0811 19:23:49.579143 139809716162560 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1920-2z4i0: server_target=xds:///psm-grpc-server:9055 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 19:23:49.579417 139809716162560 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1920-2z4i0%22;timeRange=2023-08-11T19:23:49.579227Z%2F2023-08-11T19:53:49.579227Z?project=grpc-testing
I0811 19:23:49.581465 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1920-2z4i0
labels:
name: psm-interop-client-20230811-1920-2z4i0
owner: xds-k8s-interop-test
...

I0811 19:23:49.719808 139809716162560 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1920-2z4i0 created
I0811 19:23:49.720045 139809716162560 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1920-2z4i0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:23:50.390808 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1920-2z4i0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:23:50.452530 139809716162560 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:23:50.460618 139809716162560 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1920-2z4i0
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-c5fii3u9
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-c5fii3u9
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-c5fii3u9
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:9055"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:23:50.542605 139809716162560 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:23:50.542843 139809716162560 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:23:50.653624 139809716162560 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-758f897489-gs4tn']
I0811 19:23:50.653903 139809716162560 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-758f897489-gs4tn to start
I0811 19:23:53.881279 139809716162560 k8s_base_runner.py:554] Pod psm-grpc-client-758f897489-gs4tn ready, IP: 10.120.10.252
I0811 19:23:53.881563 139809716162560 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-758f897489-gs4tn to failover_test/test_app_logs/psm-interop-client-20230811-1920-2z4i0_psm-grpc-client-758f897489-gs4tn.log
I0811 19:23:53.882001 139809592419904 k8s_log_collector.py:70] Starting log collection thread 139809592419904 for psm-grpc-client-758f897489-gs4tn
I0811 19:23:53.882143 139809716162560 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:23:55.020932 139809716162560 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:23:55.021606 139809716162560 client_app.py:200] [psm-grpc-client-758f897489-gs4tn] Waiting to report a READY channel to xds:///psm-grpc-server:9055
I0811 19:23:55.125220 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:24:05.170310 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:24:15.216179 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:24:25.261388 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:24:35.306411 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:24:51.356080 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:25:16.416187 139809716162560 client_app.py:236] [psm-grpc-client-758f897489-gs4tn] Server channel:
I0811 19:25:16.448821 139809716162560 client_app.py:249] [psm-grpc-client-758f897489-gs4tn] Found subchannel in state READY:
I0811 19:25:16.449107 139809716162560 client_app.py:211] [psm-grpc-client-758f897489-gs4tn] Channel to xds:///psm-grpc-server:9055 transitioned to state READY:
I0811 19:25:16.449272 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0811 19:25:16.449378 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0811 19:25:16.449571 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 19:25:16.484772 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0811 19:25:16.485115 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0811 19:25:16.485471 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:25:16.485706 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:20.487902 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-qkctk: 34
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-fqzxd: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-fqzxd: 33
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-qkctk: 34

I0811 19:25:20.488391 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0811 19:25:20.488518 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0811 19:25:20.488595 139809716162560 server_app.py:98] [psm-grpc-server-5bbc4d5d48-fqzxd] >> Setting health status to NOT_SERVING
I0811 19:25:20.488910 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-fqzxd:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:25:20.587476 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-fqzxd:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:25:20.619893 139809716162560 server_app.py:102] [psm-grpc-server-5bbc4d5d48-fqzxd] << Health status status: NOT_SERVING

I0811 19:25:20.620414 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:25:20.620624 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:24.647960 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-fqzxd: 33
psm-grpc-server-5bbc4d5d48-ql456: 34
psm-grpc-server-5bbc4d5d48-qkctk: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 33
psm-grpc-server-5bbc4d5d48-fqzxd: 33
psm-grpc-server-5bbc4d5d48-ql456: 34

I0811 19:25:24.648558 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-fqzxd' not found in ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456'] : Unexpected server psm-grpc-server-5bbc4d5d48-fqzxd received RPCs.
I0811 19:25:25.649903 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:25:25.650288 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:29.647299 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-fqzxd: 34
psm-grpc-server-5bbc4d5d48-qkctk: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 33
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-fqzxd: 34

I0811 19:25:29.647702 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-fqzxd' not found in ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456'] : Unexpected server psm-grpc-server-5bbc4d5d48-fqzxd received RPCs.
I0811 19:25:30.649042 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:25:30.649413 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:34.647053 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-qkctk: 47
psm-grpc-server-5bbc4d5d48-ql456: 46
psm-grpc-server-5bbc4d5d48-fqzxd: 7
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-fqzxd: 7
psm-grpc-server-5bbc4d5d48-ql456: 46
psm-grpc-server-5bbc4d5d48-qkctk: 47

I0811 19:25:34.647523 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-fqzxd' not found in ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456'] : Unexpected server psm-grpc-server-5bbc4d5d48-fqzxd received RPCs.
I0811 19:25:35.648864 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:25:35.649245 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:39.647455 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-ql456: 50
psm-grpc-server-5bbc4d5d48-qkctk: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-ql456: 50
psm-grpc-server-5bbc4d5d48-qkctk: 50

I0811 19:25:39.647908 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0811 19:25:39.648040 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0811 19:25:39.648098 139809716162560 server_app.py:98] [psm-grpc-server-5bbc4d5d48-qkctk] >> Setting health status to NOT_SERVING
I0811 19:25:39.648396 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-qkctk:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:25:39.746096 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-qkctk:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:25:39.778302 139809716162560 server_app.py:102] [psm-grpc-server-5bbc4d5d48-qkctk] << Health status status: NOT_SERVING

I0811 19:25:39.778779 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-ql456', 'psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:25:39.778973 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:43.807049 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-qkctk: 50
psm-grpc-server-5bbc4d5d48-ql456: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 50
psm-grpc-server-5bbc4d5d48-ql456: 50

I0811 19:25:43.807639 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-76d547cff6-qtpbm' not found in {'psm-grpc-server-5bbc4d5d48-qkctk': 50, 'psm-grpc-server-5bbc4d5d48-ql456': 50} : Server psm-grpc-server-alt-76d547cff6-qtpbm did not receive RPCs.
I0811 19:25:44.809022 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-ql456', 'psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:25:44.809434 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:48.807619 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-ql456: 50
psm-grpc-server-5bbc4d5d48-qkctk: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-ql456: 50
psm-grpc-server-5bbc4d5d48-qkctk: 50

I0811 19:25:48.808032 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-76d547cff6-qtpbm' not found in {'psm-grpc-server-5bbc4d5d48-ql456': 50, 'psm-grpc-server-5bbc4d5d48-qkctk': 50} : Server psm-grpc-server-alt-76d547cff6-qtpbm did not receive RPCs.
I0811 19:25:49.809381 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-ql456', 'psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:25:49.809791 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:53.838949 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-qkctk: 21
psm-grpc-server-alt-76d547cff6-qtpbm: 18
psm-grpc-server-5bbc4d5d48-ql456: 61
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-76d547cff6-qtpbm: 18
psm-grpc-server-5bbc4d5d48-qkctk: 21
psm-grpc-server-5bbc4d5d48-ql456: 61

I0811 19:25:53.839325 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-qkctk' not found in ['psm-grpc-server-5bbc4d5d48-ql456', 'psm-grpc-server-alt-76d547cff6-qtpbm'] : Unexpected server psm-grpc-server-5bbc4d5d48-qkctk received RPCs.
I0811 19:25:54.840653 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-ql456', 'psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:25:54.841086 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:25:58.879062 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 34
psm-grpc-server-5bbc4d5d48-ql456: 66
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-76d547cff6-qtpbm: 34
psm-grpc-server-5bbc4d5d48-ql456: 66

I0811 19:25:58.879412 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0811 19:25:58.879521 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0811 19:25:58.879592 139809716162560 server_app.py:98] [psm-grpc-server-5bbc4d5d48-ql456] >> Setting health status to NOT_SERVING
I0811 19:25:58.879883 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-ql456:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:25:58.976196 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-ql456:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:25:59.008160 139809716162560 server_app.py:102] [psm-grpc-server-5bbc4d5d48-ql456] << Health status status: NOT_SERVING

I0811 19:25:59.008630 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:25:59.008822 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:03.007129 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 30
psm-grpc-server-5bbc4d5d48-ql456: 70
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-76d547cff6-qtpbm: 30
psm-grpc-server-5bbc4d5d48-ql456: 70

I0811 19:26:03.007736 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-ql456' not found in ['psm-grpc-server-alt-76d547cff6-qtpbm'] : Unexpected server psm-grpc-server-5bbc4d5d48-ql456 received RPCs.
I0811 19:26:04.009040 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:26:04.009423 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:08.038700 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 76
psm-grpc-server-5bbc4d5d48-ql456: 24
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-ql456: 24
psm-grpc-server-alt-76d547cff6-qtpbm: 76

I0811 19:26:08.039155 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-ql456' not found in ['psm-grpc-server-alt-76d547cff6-qtpbm'] : Unexpected server psm-grpc-server-5bbc4d5d48-ql456 received RPCs.
I0811 19:26:09.040497 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-alt-76d547cff6-qtpbm']
I0811 19:26:09.040835 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:13.078294 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-76d547cff6-qtpbm: 100

I0811 19:26:13.078746 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0811 19:26:13.078881 139809716162560 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0811 19:26:13.078943 139809716162560 server_app.py:89] [psm-grpc-server-5bbc4d5d48-fqzxd] >> Setting health status to SERVING
I0811 19:26:13.079040 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-fqzxd:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.112903 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-fqzxd:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.145402 139809716162560 server_app.py:91] [psm-grpc-server-5bbc4d5d48-fqzxd] << Health status status: SERVING

I0811 19:26:13.145647 139809716162560 server_app.py:89] [psm-grpc-server-5bbc4d5d48-qkctk] >> Setting health status to SERVING
I0811 19:26:13.145780 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-qkctk:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.177849 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-qkctk:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.209654 139809716162560 server_app.py:91] [psm-grpc-server-5bbc4d5d48-qkctk] << Health status status: SERVING

I0811 19:26:13.209874 139809716162560 server_app.py:89] [psm-grpc-server-5bbc4d5d48-ql456] >> Setting health status to SERVING
I0811 19:26:13.210016 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-ql456:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.243814 139809716162560 grpc.py:74] [psm-grpc-server-5bbc4d5d48-ql456:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0811 19:26:13.276078 139809716162560 server_app.py:91] [psm-grpc-server-5bbc4d5d48-ql456] << Health status status: SERVING

I0811 19:26:13.276556 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:26:13.276769 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:17.318523 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-76d547cff6-qtpbm: 100

I0811 19:26:17.319029 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-fqzxd' not found in {'psm-grpc-server-alt-76d547cff6-qtpbm': 100} : Server psm-grpc-server-5bbc4d5d48-fqzxd did not receive RPCs.
I0811 19:26:18.320293 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:26:18.320584 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:22.326931 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-76d547cff6-qtpbm: 86
psm-grpc-server-5bbc4d5d48-qkctk: 14
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 14
psm-grpc-server-alt-76d547cff6-qtpbm: 86

I0811 19:26:22.327305 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-5bbc4d5d48-fqzxd' not found in {'psm-grpc-server-alt-76d547cff6-qtpbm': 86, 'psm-grpc-server-5bbc4d5d48-qkctk': 14} : Server psm-grpc-server-5bbc4d5d48-fqzxd did not receive RPCs.
I0811 19:26:23.328587 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:26:23.329030 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:27.327312 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-fqzxd: 32
psm-grpc-server-5bbc4d5d48-qkctk: 36
psm-grpc-server-alt-76d547cff6-qtpbm: 1
psm-grpc-server-5bbc4d5d48-ql456: 31
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 36
psm-grpc-server-5bbc4d5d48-fqzxd: 32
psm-grpc-server-alt-76d547cff6-qtpbm: 1
psm-grpc-server-5bbc4d5d48-ql456: 31

I0811 19:26:27.327691 139809716162560 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-76d547cff6-qtpbm' not found in ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456'] : Unexpected server psm-grpc-server-alt-76d547cff6-qtpbm received RPCs.
I0811 19:26:28.329010 139809716162560 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-5bbc4d5d48-fqzxd', 'psm-grpc-server-5bbc4d5d48-qkctk', 'psm-grpc-server-5bbc4d5d48-ql456']
I0811 19:26:28.329445 139809716162560 grpc.py:74] [psm-grpc-client-758f897489-gs4tn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:26:32.328007 139809716162560 xds_k8s_testcase.py:558] [psm-grpc-client-758f897489-gs4tn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5bbc4d5d48-fqzxd: 34
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-qkctk: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-5bbc4d5d48-qkctk: 33
psm-grpc-server-5bbc4d5d48-ql456: 33
psm-grpc-server-5bbc4d5d48-fqzxd: 34

I0811 19:26:32.328431 139809716162560 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0811 19:26:32.328575 139809716162560 xds_k8s_testcase.py:640] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0811 19:26:32.598093 139809716162560 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1920-2z4i0"
I0811 19:26:33.429811 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691781992602-602aab0b4d9f5-cc3dad82-22e5e392
I0811 19:26:44.398995 139809716162560 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1920-2z4i0"
I0811 19:26:44.568994 139809716162560 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:26:44.569199 139809716162560 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1920-2z4i0"
I0811 19:26:44.970870 139809716162560 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:26:44.971097 139809716162560 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1920-2z4i0"
I0811 19:26:45.706630 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782004974-602aab171a516-84efa5f7-8c7e1060
I0811 19:26:48.213613 139809716162560 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1920-2z4i0"
I0811 19:26:48.566290 139809716162560 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:26:48.566519 139809716162560 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1920-2z4i0"
I0811 19:26:49.213904 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782008572-602aab1a88b98-a21e47ff-1e96cfd8
I0811 19:26:51.585400 139809716162560 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1920-2z4i0"
I0811 19:26:52.110973 139809716162560 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:26:52.111158 139809716162560 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1920-2z4i0"
I0811 19:26:52.879624 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782012130-602aab1ded5a5-7427cb5b-cbd07dac
I0811 19:27:17.127233 139809716162560 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1920-2z4i0"
I0811 19:27:17.580763 139809716162560 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:27:17.581026 139809716162560 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1920-2z4i0"
I0811 19:27:18.106796 139809716162560 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1920-2z4i0" not deleted since it does not exist
I0811 19:27:18.107017 139809716162560 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1920-2z4i0"
I0811 19:27:18.796813 139809716162560 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782038212-602aab36cce62-0ef1f0d5-c9f00cdc
I0811 19:27:23.344022 139809716162560 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:27:33.540650 139809716162560 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1920-2z4i0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:27:34.357209 139809716162560 k8s_base_runner.py:487] Deleting service account psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:27:34.487593 139809716162560 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1920-2z4i0
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:28:04.697137 139809716162560 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:28:14.888119 139809716162560 k8s_base_runner.py:474] Deleting service psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:28:15.022932 139809716162560 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1920-2z4i0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:28:15.694216 139809716162560 k8s_base_runner.py:487] Deleting service account psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:28:15.832211 139809716162560 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1920-2z4i0
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:30:16.288727 139809716162560 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:30:26.748435 139809716162560 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:30:26.954092 139809716162560 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1920-2z4i0/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:30:27.608876 139809716162560 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:30:27.806780 139809716162560 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1920-2z4i0
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:32:29.388040 139809716162560 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:32:29.388373 139809716162560 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-c5fii3u9:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1920-2z4i0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-c5fii3u9%22;timeRange=2023-08-11T19:23:49.579219Z%2F2023-08-11T19:28:04.690712Z?project=grpc-testing
I0811 19:32:29.388541 139809716162560 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-97w3yjq6:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1920-2z4i0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-97w3yjq6%22;timeRange=2023-08-11T19:21:03.014041Z%2F2023-08-11T19:30:16.285776Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
----------------------------------------------------------------------
Ran 1 test in 748.275s

OK
+ for test in "${test_suites[@]}"
+ run_test outlier_detection_test
+ local test_name=outlier_detection_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test'
+ set -x
+ python -m tests.outlier_detection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:32:30.383538 140667753947136 xds_k8s_testcase.py:130] ----- Testing OutlierDetectionTest -----
I0811 19:32:30.383680 140667753947136 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:32:30.383941 140667753947136 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:32:30.399791 140667753947136 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:32:30.415326 140667753947136 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0811 19:32:30.419699 140667753947136 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1932-p9dn9
I0811 19:32:30.817670 140667753947136 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
I0811 19:32:31.127507 140667753947136 xds_k8s_testcase.py:625] Found unused xds port: 30399
I0811 19:32:31.127830 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0811 19:32:31.127915 140667753947136 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1932-p9dn9"
I0811 19:32:31.131008 140667753947136 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1932-p9dn9
type: GRPC
...

I0811 19:32:31.840460 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782351204-602aac614b15e-6ddeb231-95e70dd5
I0811 19:32:34.244455 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0811 19:32:34.244620 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0811 19:32:34.244688 140667753947136 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1932-p9dn9"
I0811 19:32:34.252118 140667753947136 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1932-p9dn9
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1932-p9dn9
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...

I0811 19:32:34.959625 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782354254-602aac6433a9c-5e99dc63-78575825
I0811 19:33:21.198149 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0811 19:33:21.198323 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0811 19:33:21.198389 140667753947136 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1932-p9dn9": psm-grpc-server:30399 -> psm-interop-backend-service-20230811-1932-p9dn9
I0811 19:33:21.209027 140667753947136 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1932-p9dn9
hostRules:
- hosts:
- psm-grpc-server:30399
pathMatcher: psm-interop-path-matcher-20230811-1932-p9dn9
name: psm-interop-url-map-20230811-1932-p9dn9
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1932-p9dn9
name: psm-interop-path-matcher-20230811-1932-p9dn9
...

I0811 19:33:21.831657 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782401211-602aac90fbb74-51caf13c-1455cb66
I0811 19:33:24.230136 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0811 19:33:24.230306 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0811 19:33:24.230386 140667753947136 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1932-p9dn9 proxy "GRPC" to URL map psm-interop-url-map-20230811-1932-p9dn9
I0811 19:33:24.232293 140667753947136 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1932-p9dn9
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1932-p9dn9
validate_for_proxyless: true
...

I0811 19:33:25.078864 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782404234-602aac93ddc81-6280bbfc-4b70bc64
I0811 19:33:27.329618 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0811 19:33:27.329857 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0811 19:33:27.329965 140667753947136 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1932-p9dn9" in network "default-vpc": 0.0.0.0:30399 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1932-p9dn9
I0811 19:33:27.332533 140667753947136 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1932-p9dn9
network: global/networks/default-vpc
portRange: 30399
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1932-p9dn9
...

I0811 19:33:27.994240 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782407334-602aac96d2be7-64e4b2c4-5fceead5
I0811 19:33:41.298283 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0811 19:33:41.298501 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0811 19:33:41.298600 140667753947136 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1932-p9dn9: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0811 19:33:41.298858 140667753947136 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1932-p9dn9%22;timeRange=2023-08-11T19:33:41.298675Z%2F2023-08-11T20:03:41.298675Z?project=grpc-testing
I0811 19:33:41.303915 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1932-p9dn9
labels:
name: psm-interop-server-20230811-1932-p9dn9
owner: xds-k8s-interop-test
...

I0811 19:33:41.459403 140667753947136 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1932-p9dn9 created
I0811 19:33:41.462373 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1932-p9dn9
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1932-p9dn9-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:33:41.536470 140667753947136 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:33:41.536710 140667753947136 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:33:41.634907 140667753947136 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1932-p9dn9-psm-grpc-server in zones=['us-central1-a']
I0811 19:33:41.635133 140667753947136 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1932-p9dn9/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:33:42.414062 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1932-p9dn9
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:33:42.473636 140667753947136 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:33:42.481597 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1932-p9dn9
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qkaahuz7
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qkaahuz7
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qkaahuz7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1932-p9dn9-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:33:42.550043 140667753947136 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:33:42.550268 140667753947136 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0811 19:33:42.737351 140667753947136 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:33:42.737823 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-58f5f48648-ctqd2 to start
I0811 19:33:45.973315 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-server-58f5f48648-ctqd2 ready, IP: 10.120.9.227
I0811 19:33:45.973578 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-58f5f48648-ctqd2 to outlier_detection_test/test_app_logs/psm-interop-server-20230811-1932-p9dn9_psm-grpc-server-58f5f48648-ctqd2.log
I0811 19:33:45.973993 140667662673472 k8s_log_collector.py:70] Starting log collection thread 140667662673472 for psm-grpc-server-58f5f48648-ctqd2
I0811 19:33:45.974125 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-58f5f48648-ksjwn to start
I0811 19:33:46.148181 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-server-58f5f48648-ksjwn ready, IP: 10.120.17.233
I0811 19:33:46.148445 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-58f5f48648-ksjwn to outlier_detection_test/test_app_logs/psm-interop-server-20230811-1932-p9dn9_psm-grpc-server-58f5f48648-ksjwn.log
I0811 19:33:46.148815 140667654280768 k8s_log_collector.py:70] Starting log collection thread 140667654280768 for psm-grpc-server-58f5f48648-ksjwn
I0811 19:33:46.148963 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-58f5f48648-l6xtw to start
I0811 19:33:46.322428 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-server-58f5f48648-l6xtw ready, IP: 10.120.10.253
I0811 19:33:46.322690 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-58f5f48648-l6xtw to outlier_detection_test/test_app_logs/psm-interop-server-20230811-1932-p9dn9_psm-grpc-server-58f5f48648-l6xtw.log
I0811 19:33:46.323071 140667645888064 k8s_log_collector.py:70] Starting log collection thread 140667645888064 for psm-grpc-server-58f5f48648-l6xtw
I0811 19:33:46.323232 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-58f5f48648-lbb4c to start
I0811 19:33:46.429188 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-server-58f5f48648-lbb4c ready, IP: 10.120.13.243
I0811 19:33:46.429445 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-58f5f48648-lbb4c to outlier_detection_test/test_app_logs/psm-interop-server-20230811-1932-p9dn9_psm-grpc-server-58f5f48648-lbb4c.log
I0811 19:33:46.429826 140667637495360 k8s_log_collector.py:70] Starting log collection thread 140667637495360 for psm-grpc-server-58f5f48648-lbb4c
I0811 19:33:46.429961 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-58f5f48648-zzj9s to start
I0811 19:33:46.603579 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-server-58f5f48648-zzj9s ready, IP: 10.120.11.236
I0811 19:33:46.603825 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-58f5f48648-zzj9s to outlier_detection_test/test_app_logs/psm-interop-server-20230811-1932-p9dn9_psm-grpc-server-58f5f48648-zzj9s.log
I0811 19:33:46.604240 140667426698816 k8s_log_collector.py:70] Starting log collection thread 140667426698816 for psm-grpc-server-58f5f48648-zzj9s
I0811 19:33:46.604392 140667753947136 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0811 19:33:53.978984 140667753947136 k8s_base_runner.py:522] Deployment psm-grpc-server has 5 replicas available
I0811 19:33:53.979697 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0811 19:33:53.979861 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0811 19:33:54.015710 140667753947136 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:33:54.266764 140667753947136 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1932-p9dn9-psm-grpc-server" in zone us-central1-a
I0811 19:33:54.266993 140667753947136 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1932-p9dn9: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1932-p9dn9-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1932-p9dn9-psm-grpc-server', zone='us-central1-a')}
I0811 19:33:54.270989 140667753947136 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1932-p9dn9-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:33:55.255773 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782434273-602aacb08388a-23cb111e-95ae796e
I0811 19:34:52.221737 140667753947136 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1932-p9dn9 to report all backends healthy: ['psm-interop-server-20230811-1932-p9dn9-psm-grpc-server']
I0811 19:35:14.144053 140667753947136 compute.py:487] Backend psm-interop-server-20230811-1932-p9dn9-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:35:18.148710 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0811 19:35:18.148923 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0811 19:35:18.149049 140667753947136 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1932-p9dn9: server_target=xds:///psm-grpc-server:30399 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0811 19:35:18.149252 140667753947136 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1932-p9dn9%22;timeRange=2023-08-11T19:35:18.149126Z%2F2023-08-11T20:05:18.149126Z?project=grpc-testing
I0811 19:35:18.151110 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1932-p9dn9
labels:
name: psm-interop-client-20230811-1932-p9dn9
owner: xds-k8s-interop-test
...

I0811 19:35:18.191497 140667753947136 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1932-p9dn9 created
I0811 19:35:18.191709 140667753947136 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1932-p9dn9/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:35:18.990581 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1932-p9dn9
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:35:19.049205 140667753947136 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:35:19.056778 140667753947136 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1932-p9dn9
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wapgvkcz
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wapgvkcz
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-wapgvkcz
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:30399"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:35:19.135060 140667753947136 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:35:19.135282 140667753947136 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:35:19.247807 140667753947136 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-686bcd8746-wdh2v']
I0811 19:35:19.248131 140667753947136 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-686bcd8746-wdh2v to start
I0811 19:35:22.479541 140667753947136 k8s_base_runner.py:554] Pod psm-grpc-client-686bcd8746-wdh2v ready, IP: 10.120.13.244
I0811 19:35:22.479788 140667753947136 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-686bcd8746-wdh2v to outlier_detection_test/test_app_logs/psm-interop-client-20230811-1932-p9dn9_psm-grpc-client-686bcd8746-wdh2v.log
I0811 19:35:22.480206 140667418306112 k8s_log_collector.py:70] Starting log collection thread 140667418306112 for psm-grpc-client-686bcd8746-wdh2v
I0811 19:35:22.480346 140667753947136 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:35:23.693577 140667753947136 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:35:23.694158 140667753947136 client_app.py:200] [psm-grpc-client-686bcd8746-wdh2v] Waiting to report a READY channel to xds:///psm-grpc-server:30399
I0811 19:35:23.797142 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:35:33.842135 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:35:43.886873 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:35:53.931559 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:36:03.977025 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:36:20.026776 140667753947136 client_app.py:236] [psm-grpc-client-686bcd8746-wdh2v] Server channel:
I0811 19:36:20.060169 140667753947136 client_app.py:249] [psm-grpc-client-686bcd8746-wdh2v] Found subchannel in state READY:
I0811 19:36:20.060458 140667753947136 client_app.py:211] [psm-grpc-client-686bcd8746-wdh2v] Channel to xds:///psm-grpc-server:30399 transitioned to state READY:
I0811 19:36:20.060569 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0811 19:36:20.060627 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0811 19:36:20.060783 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 19:36:20.096353 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0811 19:36:20.096585 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0811 19:36:20.096901 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:20.097149 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:21.127958 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-zzj9s: 20
psm-grpc-server-58f5f48648-ksjwn: 20
psm-grpc-server-58f5f48648-l6xtw: 20
psm-grpc-server-58f5f48648-ctqd2: 20
psm-grpc-server-58f5f48648-lbb4c: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 20
psm-grpc-server-58f5f48648-lbb4c: 20
psm-grpc-server-58f5f48648-ctqd2: 20
psm-grpc-server-58f5f48648-zzj9s: 20
psm-grpc-server-58f5f48648-l6xtw: 20

I0811 19:36:21.128437 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0811 19:36:21.128568 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0811 19:36:21.128821 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-58f5f48648-ctqd2 error-code-2'}]}), timeout=5, wait_for_ready=True)
I0811 19:36:21.162569 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:21.162881 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:22.197610 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 4
rpcs_by_peer:
psm-grpc-server-58f5f48648-lbb4c: 24
psm-grpc-server-58f5f48648-zzj9s: 24
psm-grpc-server-58f5f48648-ksjwn: 23
psm-grpc-server-58f5f48648-l6xtw: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 23
psm-grpc-server-58f5f48648-zzj9s: 24
psm-grpc-server-58f5f48648-lbb4c: 24
psm-grpc-server-58f5f48648-l6xtw: 25

I0811 19:36:22.198180 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 4 not less than or equal to 0 : Expected all RPCs to succeed: 4 of 100 failed.
I0811 19:36:23.199540 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:23.199882 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:24.237326 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25

I0811 19:36:24.237704 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0811 19:36:24.237810 140667753947136 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0811 19:36:24.237943 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), timeout=5, wait_for_ready=True)
I0811 19:36:24.271045 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:24.271405 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:25.307191 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-lbb4c: 25

I0811 19:36:25.307770 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:26.309101 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:26.309444 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:27.347391 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25

I0811 19:36:27.347775 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:28.349125 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:28.349503 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:29.386996 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25

I0811 19:36:29.387406 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:30.388769 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:30.389149 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:31.416921 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25

I0811 19:36:31.417347 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:32.418672 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:32.419000 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:33.446537 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-zzj9s: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25

I0811 19:36:33.446920 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:34.448282 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:34.448648 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:35.477649 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-ksjwn: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25

I0811 19:36:35.478065 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:36.479397 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:36.479743 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:37.516752 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-ksjwn: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25

I0811 19:36:37.517198 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:38.518569 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:38.518930 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:39.547027 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-l6xtw: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25

I0811 19:36:39.547463 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:40.548777 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:40.549115 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:41.576485 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-zzj9s: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25

I0811 19:36:41.576896 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:42.578229 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:42.578619 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:43.606640 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25

I0811 19:36:43.607036 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:44.607964 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:44.608369 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:45.637656 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-l6xtw: 25

I0811 19:36:45.638028 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:46.639306 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:46.639643 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:47.676722 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-ksjwn: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-l6xtw: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25

I0811 19:36:47.677193 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-ksjwn': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:48.678506 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:48.678849 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:49.706820 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-l6xtw: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-zzj9s: 25
psm-grpc-server-58f5f48648-ksjwn: 25
psm-grpc-server-58f5f48648-lbb4c: 25
psm-grpc-server-58f5f48648-l6xtw: 25

I0811 19:36:49.707202 140667753947136 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-58f5f48648-ctqd2' not found in {'psm-grpc-server-58f5f48648-ksjwn': 25, 'psm-grpc-server-58f5f48648-lbb4c': 25, 'psm-grpc-server-58f5f48648-zzj9s': 25, 'psm-grpc-server-58f5f48648-l6xtw': 25} : Server psm-grpc-server-58f5f48648-ctqd2 did not receive RPCs.
I0811 19:36:50.708495 140667753947136 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-58f5f48648-ctqd2', 'psm-grpc-server-58f5f48648-ksjwn', 'psm-grpc-server-58f5f48648-l6xtw', 'psm-grpc-server-58f5f48648-lbb4c', 'psm-grpc-server-58f5f48648-zzj9s']
I0811 19:36:50.708826 140667753947136 grpc.py:74] [psm-grpc-client-686bcd8746-wdh2v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:36:51.736365 140667753947136 xds_k8s_testcase.py:558] [psm-grpc-client-686bcd8746-wdh2v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-58f5f48648-ksjwn: 24
psm-grpc-server-58f5f48648-lbb4c: 23
psm-grpc-server-58f5f48648-ctqd2: 6
psm-grpc-server-58f5f48648-zzj9s: 23
psm-grpc-server-58f5f48648-l6xtw: 24
rpcs_by_method:
UnaryCall:
psm-grpc-server-58f5f48648-lbb4c: 23
psm-grpc-server-58f5f48648-zzj9s: 23
psm-grpc-server-58f5f48648-ctqd2: 6
psm-grpc-server-58f5f48648-l6xtw: 24
psm-grpc-server-58f5f48648-ksjwn: 24

I0811 19:36:51.736743 140667753947136 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0811 19:36:51.736878 140667753947136 xds_k8s_testcase.py:640] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0811 19:36:51.942326 140667753947136 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1932-p9dn9"
I0811 19:36:52.805097 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782611946-602aad59f4abc-e3c1f9ca-cb8bf746
I0811 19:37:16.826268 140667753947136 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1932-p9dn9"
I0811 19:37:16.990799 140667753947136 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:16.991021 140667753947136 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1932-p9dn9"
I0811 19:37:17.392216 140667753947136 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:17.392471 140667753947136 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1932-p9dn9"
I0811 19:37:17.895367 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782637395-602aad7239e0c-45c051d9-9593c4ba
I0811 19:37:20.189086 140667753947136 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1932-p9dn9"
I0811 19:37:20.602554 140667753947136 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:20.602793 140667753947136 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1932-p9dn9"
I0811 19:37:21.236760 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782640608-602aad754a30b-5553c215-c4259ee1
I0811 19:37:23.640109 140667753947136 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1932-p9dn9"
I0811 19:37:24.047261 140667753947136 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:24.047505 140667753947136 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1932-p9dn9"
I0811 19:37:24.812214 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782644052-602aad78930f0-cf564a32-fa45329e
I0811 19:37:53.319486 140667753947136 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1932-p9dn9"
I0811 19:37:53.778733 140667753947136 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:53.778979 140667753947136 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1932-p9dn9"
I0811 19:37:54.161624 140667753947136 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1932-p9dn9" not deleted since it does not exist
I0811 19:37:54.161850 140667753947136 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1932-p9dn9"
I0811 19:37:54.693524 140667753947136 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782674249-602aad955f6d5-486a707c-1838e5e1
I0811 19:37:59.188355 140667753947136 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:38:09.370825 140667753947136 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1932-p9dn9/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:38:09.967905 140667753947136 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:38:10.084818 140667753947136 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1932-p9dn9
I0811 19:38:40.283535 140667753947136 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:38:50.522289 140667753947136 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:38:50.636640 140667753947136 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1932-p9dn9/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:38:51.252660 140667753947136 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:38:51.371402 140667753947136 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1932-p9dn9
I0811 19:40:51.814834 140667753947136 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:40:51.815263 140667753947136 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-wapgvkcz:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1932-p9dn9%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-wapgvkcz%22;timeRange=2023-08-11T19:35:18.149119Z%2F2023-08-11T19:38:40.275650Z?project=grpc-testing
I0811 19:40:51.815450 140667753947136 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-qkaahuz7:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1932-p9dn9%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-qkaahuz7%22;timeRange=2023-08-11T19:33:41.298667Z%2F2023-08-11T19:40:51.814674Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
----------------------------------------------------------------------
Ran 1 test in 501.435s

OK
+ for test in "${test_suites[@]}"
+ run_test remove_neg_test
+ local test_name=remove_neg_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test'
+ set -x
+ python -m tests.remove_neg_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:40:52.883643 140093451329536 xds_k8s_testcase.py:130] ----- Testing RemoveNegTest -----
I0811 19:40:52.883831 140093451329536 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:40:52.884086 140093451329536 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:40:52.900984 140093451329536 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:40:52.916489 140093451329536 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0811 19:40:52.920850 140093451329536 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1940-ui73h
I0811 19:40:53.375769 140093451329536 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
I0811 19:40:53.678560 140093451329536 xds_k8s_testcase.py:625] Found unused xds port: 27676
I0811 19:40:53.681001 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0811 19:40:53.681239 140093451329536 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1940-ui73h"
I0811 19:40:53.684409 140093451329536 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1940-ui73h
type: GRPC
...

I0811 19:40:54.310698 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782853736-602aae408b993-60841efe-5f729dc2
I0811 19:40:56.728065 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0811 19:40:56.728295 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0811 19:40:56.728390 140093451329536 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1940-ui73h"
I0811 19:40:56.735483 140093451329536 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1940-ui73h
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1940-ui73h
protocol: GRPC
...

I0811 19:40:57.511264 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782856738-602aae4368815-ff9cb674-f0ab22b9
I0811 19:41:17.098453 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0811 19:41:17.098705 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0811 19:41:17.098804 140093451329536 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1940-ui73h": psm-grpc-server:27676 -> psm-interop-backend-service-20230811-1940-ui73h
I0811 19:41:17.110073 140093451329536 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1940-ui73h
hostRules:
- hosts:
- psm-grpc-server:27676
pathMatcher: psm-interop-path-matcher-20230811-1940-ui73h
name: psm-interop-url-map-20230811-1940-ui73h
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1940-ui73h
name: psm-interop-path-matcher-20230811-1940-ui73h
...

I0811 19:41:17.751805 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782877112-602aae56d6715-416ce42e-a2166024
I0811 19:41:20.321585 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0811 19:41:20.321798 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0811 19:41:20.321899 140093451329536 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1940-ui73h proxy "GRPC" to URL map psm-interop-url-map-20230811-1940-ui73h
I0811 19:41:20.323800 140093451329536 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1940-ui73h
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1940-ui73h
validate_for_proxyless: true
...

I0811 19:41:20.808408 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782880326-602aae59e717e-9c636166-f270e787
I0811 19:41:23.115485 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0811 19:41:23.115715 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0811 19:41:23.115820 140093451329536 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1940-ui73h" in network "default-vpc": 0.0.0.0:27676 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1940-ui73h
I0811 19:41:23.118425 140093451329536 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1940-ui73h
network: global/networks/default-vpc
portRange: 27676
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1940-ui73h
...

I0811 19:41:24.248359 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782883120-602aae5c91661-c395940e-19ca1f22
I0811 19:41:37.630440 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0811 19:41:37.630677 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0811 19:41:37.630780 140093451329536 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1940-ui73h: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:41:37.631024 140093451329536 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1940-ui73h%22;timeRange=2023-08-11T19:41:37.630869Z%2F2023-08-11T20:11:37.630869Z?project=grpc-testing
I0811 19:41:37.635909 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1940-ui73h
labels:
name: psm-interop-server-20230811-1940-ui73h
owner: xds-k8s-interop-test
...

I0811 19:41:37.787366 140093451329536 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1940-ui73h created
I0811 19:41:37.790543 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1940-ui73h
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1940-ui73h-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:41:37.857433 140093451329536 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:41:37.857672 140093451329536 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:41:37.950235 140093451329536 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1940-ui73h-psm-grpc-server in zones=['us-central1-a']
I0811 19:41:37.950444 140093451329536 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1940-ui73h/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:41:38.691235 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1940-ui73h
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:41:38.755399 140093451329536 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:41:38.763720 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1940-ui73h
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u4zc011p
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u4zc011p
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u4zc011p
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1940-ui73h-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:41:38.836925 140093451329536 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:41:38.837200 140093451329536 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 19:41:38.940903 140093451329536 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-585dd5d897-nd6nc']
I0811 19:41:38.941201 140093451329536 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-585dd5d897-nd6nc to start
I0811 19:41:42.186040 140093451329536 k8s_base_runner.py:554] Pod psm-grpc-server-585dd5d897-nd6nc ready, IP: 10.120.9.229
I0811 19:41:42.186320 140093451329536 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-585dd5d897-nd6nc to remove_neg_test/test_app_logs/psm-interop-server-20230811-1940-ui73h_psm-grpc-server-585dd5d897-nd6nc.log
I0811 19:41:42.186769 140093360744000 k8s_log_collector.py:70] Starting log collection thread 140093360744000 for psm-grpc-server-585dd5d897-nd6nc
I0811 19:41:42.186909 140093451329536 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 19:41:47.561392 140093451329536 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 19:41:47.561791 140093451329536 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230811-1940-ui73h: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 19:41:47.562015 140093451329536 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1940-ui73h%22;timeRange=2023-08-11T19:41:47.561895Z%2F2023-08-11T20:11:47.561895Z?project=grpc-testing
I0811 19:41:47.623126 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1940-ui73h
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:41:47.668985 140093451329536 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0811 19:41:47.669227 140093451329536 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0811 19:41:47.768601 140093451329536 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt in zones=['us-central1-a']
I0811 19:41:47.768811 140093451329536 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1940-ui73h/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:41:48.345310 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1940-ui73h
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:41:48.384902 140093451329536 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0811 19:41:48.392468 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230811-1940-ui73h
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-3vrgj4bl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-3vrgj4bl
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-3vrgj4bl
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:41:48.442312 140093451329536 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0811 19:41:48.442555 140093451329536 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0811 19:41:48.523596 140093451329536 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-5bc54b6bff-6mtlh']
I0811 19:41:48.523877 140093451329536 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-5bc54b6bff-6mtlh to start
I0811 19:41:52.801377 140093451329536 k8s_base_runner.py:554] Pod psm-grpc-server-alt-5bc54b6bff-6mtlh ready, IP: 10.120.10.3
I0811 19:41:52.801663 140093451329536 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-5bc54b6bff-6mtlh to remove_neg_test/test_app_logs/psm-interop-server-20230811-1940-ui73h_psm-grpc-server-alt-5bc54b6bff-6mtlh.log
I0811 19:41:52.802075 140093352351296 k8s_log_collector.py:70] Starting log collection thread 140093352351296 for psm-grpc-server-alt-5bc54b6bff-6mtlh
I0811 19:41:52.802271 140093451329536 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0811 19:41:57.065756 140093451329536 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0811 19:41:57.066164 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0811 19:41:57.066289 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0811 19:41:57.102706 140093451329536 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:41:57.376137 140093451329536 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1940-ui73h-psm-grpc-server" in zone us-central1-a
I0811 19:41:57.376388 140093451329536 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1940-ui73h: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1940-ui73h-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server', zone='us-central1-a')}
I0811 19:41:57.380355 140093451329536 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:41:58.361014 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782917382-602aae7d3e1b4-7e312103-6a1f856d
I0811 19:42:50.590580 140093451329536 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1940-ui73h to report all backends healthy: ['psm-interop-server-20230811-1940-ui73h-psm-grpc-server']
I0811 19:42:59.274804 140093451329536 compute.py:487] Backend psm-interop-server-20230811-1940-ui73h-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:43:03.317080 140093451329536 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:43:03.631202 140093451329536 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt" in zone us-central1-a
I0811 19:43:03.631457 140093451329536 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1940-ui73h: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1940-ui73h-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt', zone='us-central1-a')}
I0811 19:43:03.636361 140093451329536 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0811 19:43:04.558215 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691782983638-602aaebc6ddc3-86a81db2-8306cc0f
I0811 19:43:50.618977 140093451329536 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1940-ui73h to report all backends healthy: ['psm-interop-server-20230811-1940-ui73h-psm-grpc-server', 'psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt']
I0811 19:43:50.820233 140093451329536 compute.py:487] Backend psm-interop-server-20230811-1940-ui73h-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:43:55.089519 140093451329536 compute.py:487] Backend psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt in zone us-central1-a reported healthy
I0811 19:43:59.094219 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0811 19:43:59.094460 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0811 19:43:59.094569 140093451329536 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1940-ui73h: server_target=xds:///psm-grpc-server:27676 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 19:43:59.094811 140093451329536 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1940-ui73h%22;timeRange=2023-08-11T19:43:59.094648Z%2F2023-08-11T20:13:59.094648Z?project=grpc-testing
I0811 19:43:59.096677 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1940-ui73h
labels:
name: psm-interop-client-20230811-1940-ui73h
owner: xds-k8s-interop-test
...

I0811 19:43:59.169532 140093451329536 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1940-ui73h created
I0811 19:43:59.169837 140093451329536 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1940-ui73h/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:43:59.924935 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1940-ui73h
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:43:59.987787 140093451329536 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:43:59.995852 140093451329536 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1940-ui73h
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gso9ol2c
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gso9ol2c
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gso9ol2c
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:27676"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:44:00.072564 140093451329536 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:44:00.072798 140093451329536 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:44:00.173342 140093451329536 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-b48489cb9-dqfgs']
I0811 19:44:00.173627 140093451329536 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-b48489cb9-dqfgs to start
I0811 19:44:03.406200 140093451329536 k8s_base_runner.py:554] Pod psm-grpc-client-b48489cb9-dqfgs ready, IP: 10.120.13.245
I0811 19:44:03.406452 140093451329536 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-b48489cb9-dqfgs to remove_neg_test/test_app_logs/psm-interop-client-20230811-1940-ui73h_psm-grpc-client-b48489cb9-dqfgs.log
I0811 19:44:03.406872 140093343958592 k8s_log_collector.py:70] Starting log collection thread 140093343958592 for psm-grpc-client-b48489cb9-dqfgs
I0811 19:44:03.407004 140093451329536 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:44:04.544609 140093451329536 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:44:04.545262 140093451329536 client_app.py:200] [psm-grpc-client-b48489cb9-dqfgs] Waiting to report a READY channel to xds:///psm-grpc-server:27676
I0811 19:44:04.647898 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:44:14.693397 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:44:24.739223 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:44:34.784972 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:44:44.830088 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:45:00.879478 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:45:25.939263 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:45:51.000080 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:46:16.059486 140093451329536 client_app.py:236] [psm-grpc-client-b48489cb9-dqfgs] Server channel:
I0811 19:46:16.092236 140093451329536 client_app.py:249] [psm-grpc-client-b48489cb9-dqfgs] Found subchannel in state READY:
I0811 19:46:16.092506 140093451329536 client_app.py:211] [psm-grpc-client-b48489cb9-dqfgs] Channel to xds:///psm-grpc-server:27676 transitioned to state READY:
I0811 19:46:16.092620 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0811 19:46:16.092678 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0811 19:46:16.092855 140093451329536 grpc.py:74] [psm-grpc-client-b48489cb9-dqfgs:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0811 19:46:16.128336 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0811 19:46:16.128539 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0811 19:46:16.128721 140093451329536 grpc.py:74] [psm-grpc-client-b48489cb9-dqfgs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:46:20.156323 140093451329536 xds_k8s_testcase.py:558] [psm-grpc-client-b48489cb9-dqfgs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-585dd5d897-nd6nc: 1
psm-grpc-server-alt-5bc54b6bff-6mtlh: 99
rpcs_by_method:
UnaryCall:
psm-grpc-server-585dd5d897-nd6nc: 1
psm-grpc-server-alt-5bc54b6bff-6mtlh: 99

I0811 19:46:20.156589 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0811 19:46:20.156679 140093451329536 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0811 19:46:20.157018 140093451329536 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-585dd5d897-nd6nc', 'psm-grpc-server-alt-5bc54b6bff-6mtlh']
I0811 19:46:20.157188 140093451329536 grpc.py:74] [psm-grpc-client-b48489cb9-dqfgs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:46:24.155028 140093451329536 xds_k8s_testcase.py:558] [psm-grpc-client-b48489cb9-dqfgs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-5bc54b6bff-6mtlh: 50
psm-grpc-server-585dd5d897-nd6nc: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-585dd5d897-nd6nc: 50
psm-grpc-server-alt-5bc54b6bff-6mtlh: 50

I0811 19:46:24.307088 140093451329536 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:46:24.594009 140093451329536 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-1940-ui73h-psm-grpc-server-alt" in zone us-central1-a
I0811 19:46:24.594234 140093451329536 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1940-ui73h: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1940-ui73h-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server', zone='us-central1-a')}
I0811 19:46:24.598238 140093451329536 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1940-ui73h-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:46:25.599819 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783184600-602aaf7c14dd8-31f2b8c3-39bf3be9
I0811 19:47:13.351523 140093451329536 xds_k8s_testcase.py:435] Verifying RPCs go to servers ['psm-grpc-server-585dd5d897-nd6nc']
I0811 19:47:13.351857 140093451329536 grpc.py:74] [psm-grpc-client-b48489cb9-dqfgs:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0811 19:47:17.354779 140093451329536 xds_k8s_testcase.py:558] [psm-grpc-client-b48489cb9-dqfgs] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-585dd5d897-nd6nc: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-585dd5d897-nd6nc: 100

I0811 19:47:17.355266 140093451329536 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0811 19:47:17.355409 140093451329536 xds_k8s_testcase.py:640] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0811 19:47:17.481791 140093451329536 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1940-ui73h"
I0811 19:47:18.110709 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783237486-602aafae845b8-e5f39b6a-f09a0a75
I0811 19:47:31.047397 140093451329536 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1940-ui73h"
I0811 19:47:31.170191 140093451329536 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:47:31.170403 140093451329536 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1940-ui73h"
I0811 19:47:31.718994 140093451329536 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:47:31.719226 140093451329536 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1940-ui73h"
I0811 19:47:32.488015 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783251723-602aafbc1855d-bc27e657-b699275a
I0811 19:47:34.778445 140093451329536 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1940-ui73h"
I0811 19:47:35.021810 140093451329536 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:47:35.022016 140093451329536 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1940-ui73h"
I0811 19:47:35.703451 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783255027-602aafbf3effd-25f03e03-a76c7933
I0811 19:47:38.126550 140093451329536 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1940-ui73h"
I0811 19:47:38.600932 140093451329536 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:47:38.601171 140093451329536 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1940-ui73h"
I0811 19:47:39.375317 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783258605-602aafc2a897e-7f03a103-2a16449f
I0811 19:48:05.359442 140093451329536 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1940-ui73h"
I0811 19:48:05.758382 140093451329536 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:48:05.758601 140093451329536 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1940-ui73h"
I0811 19:48:06.223009 140093451329536 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1940-ui73h" not deleted since it does not exist
I0811 19:48:06.223230 140093451329536 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1940-ui73h"
I0811 19:48:06.802534 140093451329536 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783286319-602aafdd1697c-1cd60b8a-533d3c37
I0811 19:48:11.535124 140093451329536 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:48:21.707870 140093451329536 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1940-ui73h/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:48:22.409355 140093451329536 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:48:22.533166 140093451329536 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1940-ui73h
I0811 19:48:52.760574 140093451329536 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:49:02.948654 140093451329536 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:49:03.067703 140093451329536 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1940-ui73h/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:49:03.781128 140093451329536 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:49:03.893059 140093451329536 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1940-ui73h
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/exceptions.py:91: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
self.headers = http_resp.getheaders()
I0811 19:51:34.371562 140093451329536 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
I0811 19:51:34.429013 140093451329536 k8s_base_runner.py:466] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0811 19:51:34.429242 140093451329536 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
I0811 19:51:34.502444 140093451329536 k8s_base_runner.py:478] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0811 19:51:34.502703 140093451329536 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1940-ui73h/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:51:35.107127 140093451329536 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
I0811 19:51:35.164721 140093451329536 k8s_base_runner.py:491] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0811 19:51:35.165007 140093451329536 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1940-ui73h
I0811 19:51:35.223928 140093451329536 k8s_base_runner.py:503] Namespace psm-interop-server-20230811-1940-ui73h deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230811-1940-ui73h" not found
I0811 19:51:35.224361 140093451329536 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 19:51:35.224622 140093451329536 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-gso9ol2c:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1940-ui73h%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-gso9ol2c%22;timeRange=2023-08-11T19:43:59.094640Z%2F2023-08-11T19:48:52.758132Z?project=grpc-testing
I0811 19:51:35.224767 140093451329536 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-u4zc011p:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1940-ui73h%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-u4zc011p%22;timeRange=2023-08-11T19:41:37.630861Z%2F2023-08-11T19:51:34.371347Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
----------------------------------------------------------------------
Ran 1 test in 642.344s

OK
+ for test in "${test_suites[@]}"
+ run_test round_robin_test
+ local test_name=round_robin_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test'
+ set -x
+ python -m tests.round_robin_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 19:51:36.220458 140085689835520 xds_k8s_testcase.py:130] ----- Testing RoundRobinTest -----
I0811 19:51:36.220600 140085689835520 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 19:51:36.220836 140085689835520 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 19:51:36.236865 140085689835520 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 19:51:36.252663 140085689835520 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0811 19:51:36.257049 140085689835520 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230811-1951-54r1q
I0811 19:51:36.589358 140085689835520 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
I0811 19:51:36.978821 140085689835520 xds_k8s_testcase.py:625] Found unused xds port: 61738
I0811 19:51:36.979171 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0811 19:51:36.979293 140085689835520 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-1951-54r1q"
I0811 19:51:36.982423 140085689835520 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230811-1951-54r1q
type: GRPC
...

I0811 19:51:37.621116 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783497048-602ab0a60e1a9-38c24448-b298da99
I0811 19:51:39.993769 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0811 19:51:39.994005 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0811 19:51:39.994101 140085689835520 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-1951-54r1q"
I0811 19:51:40.001417 140085689835520 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-1951-54r1q
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-1951-54r1q
protocol: GRPC
...

I0811 19:51:40.595902 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783500003-602ab0a8df89b-c32f61a4-d10ea1f5
I0811 19:52:33.210564 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0811 19:52:33.210803 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0811 19:52:33.210895 140085689835520 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-1951-54r1q": psm-grpc-server:61738 -> psm-interop-backend-service-20230811-1951-54r1q
I0811 19:52:33.221912 140085689835520 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1951-54r1q
hostRules:
- hosts:
- psm-grpc-server:61738
pathMatcher: psm-interop-path-matcher-20230811-1951-54r1q
name: psm-interop-url-map-20230811-1951-54r1q
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-1951-54r1q
name: psm-interop-path-matcher-20230811-1951-54r1q
...

I0811 19:52:34.061074 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783553224-602ab0dba0e8f-bd282e9a-c6721bc7
I0811 19:52:36.451080 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0811 19:52:36.451285 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0811 19:52:36.451415 140085689835520 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-1951-54r1q proxy "GRPC" to URL map psm-interop-url-map-20230811-1951-54r1q
I0811 19:52:36.453359 140085689835520 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-1951-54r1q
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-1951-54r1q
validate_for_proxyless: true
...

I0811 19:52:36.832365 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783556455-602ab0deb5cb7-160f215c-d0ea07f2
I0811 19:52:39.264118 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0811 19:52:39.264346 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0811 19:52:39.264452 140085689835520 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-1951-54r1q" in network "default-vpc": 0.0.0.0:61738 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1951-54r1q
I0811 19:52:39.266988 140085689835520 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-1951-54r1q
network: global/networks/default-vpc
portRange: 61738
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-1951-54r1q
...

I0811 19:52:40.213531 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783559268-602ab0e164a83-20795af1-ac293899
I0811 19:52:53.538155 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0811 19:52:53.538389 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0811 19:52:53.538507 140085689835520 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-1951-54r1q: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0811 19:52:53.538750 140085689835520 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1951-54r1q%22;timeRange=2023-08-11T19:52:53.538591Z%2F2023-08-11T20:22:53.538591Z?project=grpc-testing
I0811 19:52:53.543786 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-1951-54r1q
labels:
name: psm-interop-server-20230811-1951-54r1q
owner: xds-k8s-interop-test
...

I0811 19:52:53.679914 140085689835520 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-1951-54r1q created
I0811 19:52:53.682870 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1951-54r1q
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-1951-54r1q-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 19:52:53.752896 140085689835520 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 19:52:53.753169 140085689835520 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 19:52:53.844796 140085689835520 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-1951-54r1q-psm-grpc-server in zones=['us-central1-a']
I0811 19:52:53.845136 140085689835520 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1951-54r1q/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:52:54.625182 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1951-54r1q
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:52:54.690853 140085689835520 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 19:52:54.699289 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-1951-54r1q
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1beifxu7
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1beifxu7
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-1beifxu7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-1951-54r1q-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:52:54.770442 140085689835520 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 19:52:54.770666 140085689835520 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0811 19:52:54.916054 140085689835520 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-856d9dfcd9-wnlb7', 'psm-grpc-server-856d9dfcd9-zftc6']
I0811 19:52:54.916411 140085689835520 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-856d9dfcd9-wnlb7 to start
I0811 19:52:58.153238 140085689835520 k8s_base_runner.py:554] Pod psm-grpc-server-856d9dfcd9-wnlb7 ready, IP: 10.120.16.79
I0811 19:52:58.153506 140085689835520 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-856d9dfcd9-wnlb7 to round_robin_test/test_app_logs/psm-interop-server-20230811-1951-54r1q_psm-grpc-server-856d9dfcd9-wnlb7.log
I0811 19:52:58.153952 140085599184448 k8s_log_collector.py:70] Starting log collection thread 140085599184448 for psm-grpc-server-856d9dfcd9-wnlb7
I0811 19:52:58.154084 140085689835520 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-856d9dfcd9-zftc6 to start
I0811 19:52:58.262062 140085689835520 k8s_base_runner.py:554] Pod psm-grpc-server-856d9dfcd9-zftc6 ready, IP: 10.120.14.169
I0811 19:52:58.262306 140085689835520 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-856d9dfcd9-zftc6 to round_robin_test/test_app_logs/psm-interop-server-20230811-1951-54r1q_psm-grpc-server-856d9dfcd9-zftc6.log
I0811 19:52:58.262670 140085590791744 k8s_log_collector.py:70] Starting log collection thread 140085590791744 for psm-grpc-server-856d9dfcd9-zftc6
I0811 19:52:58.262801 140085689835520 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0811 19:53:03.561458 140085689835520 k8s_base_runner.py:522] Deployment psm-grpc-server has 2 replicas available
I0811 19:53:03.561928 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0811 19:53:03.562046 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0811 19:53:03.598471 140085689835520 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 19:53:03.745148 140085689835520 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-1951-54r1q-psm-grpc-server" in zone us-central1-a
I0811 19:53:03.745392 140085689835520 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-1951-54r1q: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-1951-54r1q-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1951-54r1q-psm-grpc-server', zone='us-central1-a')}
I0811 19:53:03.749353 140085689835520 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-1951-54r1q-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 19:53:04.704440 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783583751-602ab0f8bdc6f-f985ef37-3ada5014
I0811 19:53:57.036641 140085689835520 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-1951-54r1q to report all backends healthy: ['psm-interop-server-20230811-1951-54r1q-psm-grpc-server']
I0811 19:54:06.060528 140085689835520 compute.py:487] Backend psm-interop-server-20230811-1951-54r1q-psm-grpc-server in zone us-central1-a reported healthy
I0811 19:54:10.065181 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0811 19:54:10.065394 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0811 19:54:10.065527 140085689835520 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-1951-54r1q: server_target=xds:///psm-grpc-server:61738 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 19:54:10.065747 140085689835520 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1951-54r1q%22;timeRange=2023-08-11T19:54:10.065613Z%2F2023-08-11T20:24:10.065613Z?project=grpc-testing
I0811 19:54:10.067600 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-1951-54r1q
labels:
name: psm-interop-client-20230811-1951-54r1q
owner: xds-k8s-interop-test
...

I0811 19:54:10.147960 140085689835520 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-1951-54r1q created
I0811 19:54:10.148179 140085689835520 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1951-54r1q/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:54:10.775738 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1951-54r1q
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 19:54:10.844509 140085689835520 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 19:54:10.852158 140085689835520 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-1951-54r1q
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xz3nyz9a
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xz3nyz9a
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xz3nyz9a
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:61738"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 19:54:10.930965 140085689835520 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 19:54:10.931206 140085689835520 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 19:54:11.033805 140085689835520 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-9d6875d56-zbhwh']
I0811 19:54:11.034078 140085689835520 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-9d6875d56-zbhwh to start
I0811 19:54:14.265815 140085689835520 k8s_base_runner.py:554] Pod psm-grpc-client-9d6875d56-zbhwh ready, IP: 10.120.13.247
I0811 19:54:14.266059 140085689835520 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-9d6875d56-zbhwh to round_robin_test/test_app_logs/psm-interop-client-20230811-1951-54r1q_psm-grpc-client-9d6875d56-zbhwh.log
I0811 19:54:14.266463 140085582399040 k8s_log_collector.py:70] Starting log collection thread 140085582399040 for psm-grpc-client-9d6875d56-zbhwh
I0811 19:54:14.266595 140085689835520 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 19:54:15.476771 140085689835520 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 19:54:15.477408 140085689835520 client_app.py:200] [psm-grpc-client-9d6875d56-zbhwh] Waiting to report a READY channel to xds:///psm-grpc-server:61738
I0811 19:54:15.578702 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:54:25.624190 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:54:35.668272 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:54:45.712761 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:54:55.756940 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:55:11.806581 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:55:36.867711 140085689835520 client_app.py:236] [psm-grpc-client-9d6875d56-zbhwh] Server channel:
I0811 19:55:36.900123 140085689835520 client_app.py:249] [psm-grpc-client-9d6875d56-zbhwh] Found subchannel in state READY:
I0811 19:55:36.900453 140085689835520 client_app.py:211] [psm-grpc-client-9d6875d56-zbhwh] Channel to xds:///psm-grpc-server:61738 transitioned to state READY:
I0811 19:55:36.900598 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0811 19:55:36.900675 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0811 19:55:36.900853 140085689835520 grpc.py:74] [psm-grpc-client-9d6875d56-zbhwh:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0811 19:55:36.935456 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0811 19:55:36.935686 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0811 19:55:36.935878 140085689835520 grpc.py:74] [psm-grpc-client-9d6875d56-zbhwh:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:55:40.964474 140085689835520 xds_k8s_testcase.py:558] [psm-grpc-client-9d6875d56-zbhwh] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-856d9dfcd9-zftc6: 50
psm-grpc-server-856d9dfcd9-wnlb7: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-856d9dfcd9-wnlb7: 50
psm-grpc-server-856d9dfcd9-zftc6: 50

I0811 19:55:40.964742 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0811 19:55:40.964857 140085689835520 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0811 19:55:40.965050 140085689835520 grpc.py:74] [psm-grpc-client-9d6875d56-zbhwh:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 19:55:44.964766 140085689835520 xds_k8s_testcase.py:558] [psm-grpc-client-9d6875d56-zbhwh] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-856d9dfcd9-wnlb7: 50
psm-grpc-server-856d9dfcd9-zftc6: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-856d9dfcd9-wnlb7: 50
psm-grpc-server-856d9dfcd9-zftc6: 50

I0811 19:55:44.965129 140085689835520 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0811 19:55:44.965270 140085689835520 xds_k8s_testcase.py:640] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0811 19:55:45.074079 140085689835520 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-1951-54r1q"
I0811 19:55:45.842454 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783745077-602ab19298274-2c23ea00-d9f0e707
I0811 19:56:16.653540 140085689835520 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-1951-54r1q"
I0811 19:56:16.820416 140085689835520 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:56:16.820633 140085689835520 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-1951-54r1q"
I0811 19:56:17.289945 140085689835520 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:56:17.290169 140085689835520 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-1951-54r1q"
I0811 19:56:18.044697 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783777292-602ab1b1512a7-97ca7b3a-4e9f233e
I0811 19:56:20.427094 140085689835520 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-1951-54r1q"
I0811 19:56:20.922525 140085689835520 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:56:20.922739 140085689835520 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-1951-54r1q"
I0811 19:56:21.724224 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783780928-602ab1b4c8cf0-1c4ee7a5-566bd1fe
I0811 19:56:24.008012 140085689835520 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-1951-54r1q"
I0811 19:56:24.499658 140085689835520 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:56:24.499882 140085689835520 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-1951-54r1q"
I0811 19:56:25.167489 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783784505-602ab1b832068-660dd700-7f8d5e81
I0811 19:57:04.635390 140085689835520 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-1951-54r1q"
I0811 19:57:05.013673 140085689835520 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:57:05.013980 140085689835520 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-1951-54r1q"
I0811 19:57:05.536240 140085689835520 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-1951-54r1q" not deleted since it does not exist
I0811 19:57:05.536480 140085689835520 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-1951-54r1q"
I0811 19:57:06.096580 140085689835520 compute.py:589] Waiting 600 sec for compute operation id: operation-1691783825638-602ab1df6c460-e3b9b143-7d732d72
I0811 19:57:10.603435 140085689835520 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 19:57:20.793558 140085689835520 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-1951-54r1q/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:57:21.521785 140085689835520 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 19:57:21.643555 140085689835520 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-1951-54r1q
I0811 19:57:51.832531 140085689835520 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 19:58:02.003242 140085689835520 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 19:58:02.126193 140085689835520 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-1951-54r1q/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 19:58:02.800418 140085689835520 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 19:58:02.915952 140085689835520 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-1951-54r1q
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:00:03.364897 140085689835520 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0811 20:00:03.365310 140085689835520 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-xz3nyz9a:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-1951-54r1q%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-xz3nyz9a%22;timeRange=2023-08-11T19:54:10.065605Z%2F2023-08-11T19:57:51.828456Z?project=grpc-testing
I0811 20:00:03.365479 140085689835520 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-1beifxu7:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-1951-54r1q%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-1beifxu7%22;timeRange=2023-08-11T19:52:53.538583Z%2F2023-08-11T20:00:03.364739Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
----------------------------------------------------------------------
Ran 1 test in 507.148s

OK
+ for test in "${test_suites[@]}"
+ run_test bootstrap_generator_test
+ local test_name=bootstrap_generator_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test'
+ set -x
+ python -m tests.bootstrap_generator_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --client_image=gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4 --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/bin/python
I0811 20:00:04.401910 140336156241920 xds_k8s_testcase.py:130] ----- Testing BootstrapGeneratorClientTest -----
I0811 20:00:04.402082 140336156241920 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 20:00:04.402419 140336156241920 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 20:00:04.419023 140336156241920 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 20:00:04.434700 140336156241920 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0811 20:00:04.439045 140336156241920 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20230811-2000-rtgkr
I0811 20:00:05.045725 140336156241920 bootstrap_generator_testcase.py:88] Found unused xds port: 38256
I0811 20:00:05.046196 140336156241920 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-2000-rtgkr"
I0811 20:00:05.049546 140336156241920 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20230811-2000-rtgkr
type: GRPC
...

I0811 20:00:05.736071 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784005114-602ab28a95c20-aabadf08-ededffac
I0811 20:00:08.125098 140336156241920 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-2000-rtgkr"
I0811 20:00:08.132457 140336156241920 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-2000-rtgkr
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-2000-rtgkr
protocol: GRPC
...

I0811 20:00:09.036435 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784008134-602ab28d76edd-230ee15d-80e49834
I0811 20:00:30.946363 140336156241920 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-2000-rtgkr": psm-grpc-server:38256 -> psm-interop-backend-service-20230811-2000-rtgkr
I0811 20:00:30.957584 140336156241920 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-2000-rtgkr
hostRules:
- hosts:
- psm-grpc-server:38256
pathMatcher: psm-interop-path-matcher-20230811-2000-rtgkr
name: psm-interop-url-map-20230811-2000-rtgkr
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-2000-rtgkr
name: psm-interop-path-matcher-20230811-2000-rtgkr
...

I0811 20:00:31.837602 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784030959-602ab2a33b84c-42cd8b3d-573f4bbf
I0811 20:00:34.200590 140336156241920 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-2000-rtgkr proxy "GRPC" to URL map psm-interop-url-map-20230811-2000-rtgkr
I0811 20:00:34.202703 140336156241920 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-2000-rtgkr
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-2000-rtgkr
validate_for_proxyless: true
...

I0811 20:00:34.919885 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784034204-602ab2a653ba7-aa489e80-609459fc
I0811 20:00:37.239887 140336156241920 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-2000-rtgkr" in network "default-vpc": 0.0.0.0:38256 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-2000-rtgkr
I0811 20:00:37.242587 140336156241920 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-2000-rtgkr
network: global/networks/default-vpc
portRange: 38256
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-2000-rtgkr
...

I0811 20:00:38.561394 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784037246-602ab2a93a4fb-34151c63-ae595a4f
I0811 20:01:06.690441 140336156241920 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
I0811 20:01:06.815467 140336156241920 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-2000-rtgkr: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 20:01:06.815843 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-2000-rtgkr%22;timeRange=2023-08-11T20:01:06.815687Z%2F2023-08-11T20:31:06.815687Z?project=grpc-testing
I0811 20:01:06.820694 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-2000-rtgkr
labels:
name: psm-interop-server-20230811-2000-rtgkr
owner: xds-k8s-interop-test
...

I0811 20:01:06.953391 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-2000-rtgkr created
I0811 20:01:06.956436 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-2000-rtgkr-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 20:01:07.034043 140336156241920 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 20:01:07.034254 140336156241920 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 20:01:07.140976 140336156241920 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-2000-rtgkr-psm-grpc-server in zones=['us-central1-a']
I0811 20:01:07.141207 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2000-rtgkr/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:01:07.884633 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:01:07.952227 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 20:01:07.960015 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2000-rtgkr
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xqtktts1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xqtktts1
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xqtktts1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-2000-rtgkr-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:01:08.043546 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 20:01:08.043769 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 20:01:08.169240 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-b48bc74fc-gtvc7']
I0811 20:01:08.169528 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-b48bc74fc-gtvc7 to start
I0811 20:01:11.431782 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-server-b48bc74fc-gtvc7 ready, IP: 10.120.11.241
I0811 20:01:11.432025 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-b48bc74fc-gtvc7 to bootstrap_generator_test/test_app_logs/psm-interop-server-20230811-2000-rtgkr_psm-grpc-server-b48bc74fc-gtvc7.log
I0811 20:01:11.432421 140336065193536 k8s_log_collector.py:70] Starting log collection thread 140336065193536 for psm-grpc-server-b48bc74fc-gtvc7
I0811 20:01:11.432555 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 20:01:17.886707 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 20:01:17.928851 140336156241920 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:01:18.207530 140336156241920 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-2000-rtgkr-psm-grpc-server" in zone us-central1-a
I0811 20:01:18.207742 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2000-rtgkr: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-2000-rtgkr-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2000-rtgkr-psm-grpc-server', zone='us-central1-a')}
I0811 20:01:18.211618 140336156241920 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2000-rtgkr-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 20:01:19.130173 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784078213-602ab2d04c16b-c3b3ff81-a6ab1dc2
I0811 20:02:16.261855 140336156241920 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-2000-rtgkr to report all backends healthy: ['psm-interop-server-20230811-2000-rtgkr-psm-grpc-server']
I0811 20:02:33.332222 140336156241920 compute.py:487] Backend psm-interop-server-20230811-2000-rtgkr-psm-grpc-server in zone us-central1-a reported healthy
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0811 20:02:37.337118 140336156241920 bootstrap_generator_test.py:149] ----- testing bootstrap generator version v0.14.0 -----
I0811 20:02:37.339136 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2000-rtgkr: server_target=xds:///psm-grpc-server:38256 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:02:37.339459 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2000-rtgkr%22;timeRange=2023-08-11T20:02:37.339296Z%2F2023-08-11T20:32:37.339296Z?project=grpc-testing
I0811 20:02:37.341322 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2000-rtgkr
labels:
name: psm-interop-client-20230811-2000-rtgkr
owner: xds-k8s-interop-test
...

I0811 20:02:37.438359 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2000-rtgkr created
I0811 20:02:37.438585 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:02:38.257335 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:02:38.327440 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:02:38.335286 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yplewisp
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yplewisp
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yplewisp
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:38256"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:02:38.413882 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:02:38.414138 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:02:38.529183 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-55c6fb7658-b6g24']
I0811 20:02:38.529479 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-55c6fb7658-b6g24 to start
I0811 20:02:41.789485 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-55c6fb7658-b6g24 ready, IP: 10.120.10.7
I0811 20:02:41.789731 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-55c6fb7658-b6g24 to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2000-rtgkr_psm-grpc-client-55c6fb7658-b6g24.log
I0811 20:02:41.790130 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-55c6fb7658-b6g24
I0811 20:02:41.790263 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:02:43.024993 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:02:43.025575 140336156241920 client_app.py:200] [psm-grpc-client-55c6fb7658-b6g24] Waiting to report a READY channel to xds:///psm-grpc-server:38256
I0811 20:02:43.129088 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:02:53.175837 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:03:03.221090 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:03:13.266778 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:03:23.311623 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:03:39.361751 140336156241920 client_app.py:236] [psm-grpc-client-55c6fb7658-b6g24] Server channel:
I0811 20:03:39.395115 140336156241920 client_app.py:249] [psm-grpc-client-55c6fb7658-b6g24] Found subchannel in state READY:
I0811 20:03:39.395488 140336156241920 client_app.py:211] [psm-grpc-client-55c6fb7658-b6g24] Channel to xds:///psm-grpc-server:38256 transitioned to state READY:
I0811 20:03:39.395772 140336156241920 grpc.py:74] [psm-grpc-client-55c6fb7658-b6g24:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:03:39.431458 140336156241920 grpc.py:74] [psm-grpc-client-55c6fb7658-b6g24:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:03:43.434515 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-55c6fb7658-b6g24] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b48bc74fc-gtvc7: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b48bc74fc-gtvc7: 100

I0811 20:03:43.434833 140336156241920 bootstrap_generator_test.py:127] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0811 20:03:43.437488 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:03:53.605910 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:03:54.401996 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:03:54.547173 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2000-rtgkr
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0811 20:04:25.741463 140336156241920 bootstrap_generator_test.py:149] ----- testing bootstrap generator version v0.13.0 -----
I0811 20:04:25.743435 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2000-rtgkr: server_target=xds:///psm-grpc-server:38256 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:04:25.743700 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2000-rtgkr%22;timeRange=2023-08-11T20:04:25.743566Z%2F2023-08-11T20:34:25.743566Z?project=grpc-testing
I0811 20:04:25.745720 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2000-rtgkr
labels:
name: psm-interop-client-20230811-2000-rtgkr
owner: xds-k8s-interop-test
...

I0811 20:04:25.810269 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2000-rtgkr created
I0811 20:04:25.810516 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:04:26.536024 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:04:26.599298 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:04:26.607073 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zctdslyx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zctdslyx
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zctdslyx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:38256"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:04:26.679461 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:04:26.679713 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:04:26.781321 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-85d7d87c67-r4689']
I0811 20:04:26.781591 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-85d7d87c67-r4689 to start
I0811 20:04:30.011668 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-85d7d87c67-r4689 ready, IP: 10.120.13.249
I0811 20:04:30.011934 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-85d7d87c67-r4689 to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2000-rtgkr_psm-grpc-client-85d7d87c67-r4689.log
I0811 20:04:30.012307 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-85d7d87c67-r4689
I0811 20:04:30.012445 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:04:31.152390 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:04:31.153030 140336156241920 client_app.py:200] [psm-grpc-client-85d7d87c67-r4689] Waiting to report a READY channel to xds:///psm-grpc-server:38256
I0811 20:04:31.251860 140336156241920 client_app.py:236] [psm-grpc-client-85d7d87c67-r4689] Server channel:
I0811 20:04:31.284574 140336156241920 client_app.py:249] [psm-grpc-client-85d7d87c67-r4689] Found subchannel in state READY:
I0811 20:04:31.285009 140336156241920 client_app.py:211] [psm-grpc-client-85d7d87c67-r4689] Channel to xds:///psm-grpc-server:38256 transitioned to state READY:
I0811 20:04:31.285276 140336156241920 grpc.py:74] [psm-grpc-client-85d7d87c67-r4689:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:04:31.319310 140336156241920 grpc.py:74] [psm-grpc-client-85d7d87c67-r4689:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:04:35.337904 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-85d7d87c67-r4689] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b48bc74fc-gtvc7: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b48bc74fc-gtvc7: 100

I0811 20:04:35.338207 140336156241920 bootstrap_generator_test.py:127] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0811 20:04:35.341158 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
I0811 20:04:45.488220 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:04:46.296341 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:04:46.414495 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2000-rtgkr
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0811 20:05:16.574625 140336156241920 bootstrap_generator_test.py:149] ----- testing bootstrap generator version v0.12.0 -----
I0811 20:05:16.576595 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2000-rtgkr: server_target=xds:///psm-grpc-server:38256 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:05:16.576868 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2000-rtgkr%22;timeRange=2023-08-11T20:05:16.576748Z%2F2023-08-11T20:35:16.576748Z?project=grpc-testing
I0811 20:05:16.578744 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2000-rtgkr
labels:
name: psm-interop-client-20230811-2000-rtgkr
owner: xds-k8s-interop-test
...

I0811 20:05:16.619050 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2000-rtgkr created
I0811 20:05:16.619270 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:05:17.242425 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:05:17.280703 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:05:17.288356 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uwrnzg04
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uwrnzg04
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uwrnzg04
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:38256"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:05:17.339036 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:05:17.339269 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:05:17.420231 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6cbfbf75df-9jvgx']
I0811 20:05:17.420501 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-6cbfbf75df-9jvgx to start
I0811 20:05:20.654220 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-6cbfbf75df-9jvgx ready, IP: 10.120.13.250
I0811 20:05:20.654494 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-6cbfbf75df-9jvgx to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2000-rtgkr_psm-grpc-client-6cbfbf75df-9jvgx.log
I0811 20:05:20.654840 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-6cbfbf75df-9jvgx
I0811 20:05:20.654974 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:05:21.773867 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:05:21.774523 140336156241920 client_app.py:200] [psm-grpc-client-6cbfbf75df-9jvgx] Waiting to report a READY channel to xds:///psm-grpc-server:38256
I0811 20:05:21.873887 140336156241920 client_app.py:236] [psm-grpc-client-6cbfbf75df-9jvgx] Server channel:
I0811 20:05:21.907156 140336156241920 client_app.py:249] [psm-grpc-client-6cbfbf75df-9jvgx] Found subchannel in state READY:
I0811 20:05:21.907578 140336156241920 client_app.py:211] [psm-grpc-client-6cbfbf75df-9jvgx] Channel to xds:///psm-grpc-server:38256 transitioned to state READY:
I0811 20:05:21.907826 140336156241920 grpc.py:74] [psm-grpc-client-6cbfbf75df-9jvgx:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:05:21.942833 140336156241920 grpc.py:74] [psm-grpc-client-6cbfbf75df-9jvgx:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:05:25.952454 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-6cbfbf75df-9jvgx] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b48bc74fc-gtvc7: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b48bc74fc-gtvc7: 100

I0811 20:05:25.952764 140336156241920 bootstrap_generator_test.py:127] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0811 20:05:25.955530 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
I0811 20:05:36.083224 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:05:36.879203 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:05:36.953584 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2000-rtgkr
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0811 20:06:07.093285 140336156241920 bootstrap_generator_test.py:149] ----- testing bootstrap generator version v0.11.0 -----
I0811 20:06:07.095243 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2000-rtgkr: server_target=xds:///psm-grpc-server:38256 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:06:07.095519 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2000-rtgkr%22;timeRange=2023-08-11T20:06:07.095403Z%2F2023-08-11T20:36:07.095403Z?project=grpc-testing
I0811 20:06:07.097276 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2000-rtgkr
labels:
name: psm-interop-client-20230811-2000-rtgkr
owner: xds-k8s-interop-test
...

I0811 20:06:07.138202 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2000-rtgkr created
I0811 20:06:07.138448 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:06:07.825636 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:06:07.865430 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:06:07.873121 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2000-rtgkr
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-aku9yyih
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-aku9yyih
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-aku9yyih
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:38256"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:06:07.924326 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:06:07.924588 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:06:08.003410 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-758985bf67-gm25s']
I0811 20:06:08.003681 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-758985bf67-gm25s to start
I0811 20:06:10.192781 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-758985bf67-gm25s ready, IP: 10.120.13.251
I0811 20:06:10.193070 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-758985bf67-gm25s to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2000-rtgkr_psm-grpc-client-758985bf67-gm25s.log
I0811 20:06:10.193407 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-758985bf67-gm25s
I0811 20:06:10.193542 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:06:12.349947 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:06:12.350528 140336156241920 client_app.py:200] [psm-grpc-client-758985bf67-gm25s] Waiting to report a READY channel to xds:///psm-grpc-server:38256
I0811 20:06:12.450093 140336156241920 client_app.py:236] [psm-grpc-client-758985bf67-gm25s] Server channel:
I0811 20:06:12.482844 140336156241920 client_app.py:249] [psm-grpc-client-758985bf67-gm25s] Found subchannel in state READY:
I0811 20:06:12.483240 140336156241920 client_app.py:211] [psm-grpc-client-758985bf67-gm25s] Channel to xds:///psm-grpc-server:38256 transitioned to state READY:
I0811 20:06:12.483520 140336156241920 grpc.py:74] [psm-grpc-client-758985bf67-gm25s:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:06:12.517972 140336156241920 grpc.py:74] [psm-grpc-client-758985bf67-gm25s:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:06:16.516410 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-758985bf67-gm25s] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-b48bc74fc-gtvc7: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-b48bc74fc-gtvc7: 100

I0811 20:06:16.516708 140336156241920 bootstrap_generator_test.py:127] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0811 20:06:16.519462 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
I0811 20:06:26.710486 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2000-rtgkr/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:06:27.411124 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:06:27.530511 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2000-rtgkr
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0811 20:06:57.766247 140336156241920 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:06:58.068167 140336156241920 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-2000-rtgkr-psm-grpc-server" in zone us-central1-a
I0811 20:06:58.068400 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2000-rtgkr: set()
I0811 20:06:58.071794 140336156241920 compute.py:537] Patching compute resource:
---
backends: []
...

I0811 20:06:58.898574 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784418073-602ab41469d4e-c17da5e6-99f7c83c
I0811 20:07:53.791241 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 20:08:03.958876 140336156241920 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 20:08:04.057692 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2000-rtgkr/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:08:04.909976 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 20:08:05.023025 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-2000-rtgkr
I0811 20:08:35.237970 140336156241920 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-2000-rtgkr"
I0811 20:08:36.115533 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784515241-602ab47114707-ad7665ec-e62b00e6
I0811 20:08:45.172526 140336156241920 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-2000-rtgkr"
I0811 20:08:45.307547 140336156241920 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:08:45.307752 140336156241920 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-2000-rtgkr"
I0811 20:08:45.804522 140336156241920 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:08:45.804727 140336156241920 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-2000-rtgkr"
I0811 20:08:46.547244 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784525807-602ab47b28001-2fb8645a-f2ce08a8
I0811 20:08:48.877777 140336156241920 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-2000-rtgkr"
I0811 20:08:49.132689 140336156241920 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:08:49.132896 140336156241920 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-2000-rtgkr"
I0811 20:08:49.818723 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784529138-602ab47e55232-6cde3021-45d36fdd
I0811 20:08:52.178535 140336156241920 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-2000-rtgkr"
I0811 20:08:52.587430 140336156241920 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:08:52.587644 140336156241920 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-2000-rtgkr"
I0811 20:08:53.148351 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784532592-602ab481a0777-e527745e-48499882
I0811 20:09:19.482441 140336156241920 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-2000-rtgkr"
I0811 20:09:19.963198 140336156241920 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:09:19.963459 140336156241920 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-2000-rtgkr"
I0811 20:09:20.370268 140336156241920 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-2000-rtgkr" not deleted since it does not exist
I0811 20:09:20.370514 140336156241920 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-2000-rtgkr"
I0811 20:09:20.889680 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784560459-602ab49c34075-40cc4208-5fc8430d
I0811 20:09:25.389364 140336156241920 xds_k8s_testcase.py:130] ----- Testing BootstrapGeneratorServerTest -----
I0811 20:09:25.389602 140336156241920 xds_k8s_testcase.py:131] Logs timezone: UTC
I0811 20:09:25.390124 140336156241920 skips.py:124] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0811 20:09:25.405442 140336156241920 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0811 20:09:25.420662 140336156241920 k8s.py:130] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0811 20:09:25.421043 140336156241920 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20230811-2009-igj3k
I0811 20:09:26.091099 140336156241920 bootstrap_generator_testcase.py:88] Found unused xds port: 39572
I0811 20:09:26.091354 140336156241920 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230811-2009-igj3k"
I0811 20:09:26.094694 140336156241920 compute.py:530] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20230811-2009-igj3k
type: GRPC
...

I0811 20:09:26.719845 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784566160-602ab4a1a3c4d-5ce854a3-b8269551
I0811 20:09:29.101021 140336156241920 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230811-2009-igj3k"
I0811 20:09:29.107755 140336156241920 compute.py:530] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230811-2009-igj3k
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230811-2009-igj3k
protocol: GRPC
...

I0811 20:09:29.905907 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784569109-602ab4a473d1b-b32716ad-72e574f7
I0811 20:09:49.508983 140336156241920 traffic_director.py:475] Creating URL map "psm-interop-url-map-20230811-2009-igj3k": psm-grpc-server:39572 -> psm-interop-backend-service-20230811-2009-igj3k
I0811 20:09:49.519789 140336156241920 compute.py:530] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-2009-igj3k
hostRules:
- hosts:
- psm-grpc-server:39572
pathMatcher: psm-interop-path-matcher-20230811-2009-igj3k
name: psm-interop-url-map-20230811-2009-igj3k
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230811-2009-igj3k
name: psm-interop-path-matcher-20230811-2009-igj3k
...

I0811 20:09:50.426000 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784589522-602ab4b7eb530-f0daaad1-385554a1
I0811 20:09:52.969728 140336156241920 traffic_director.py:574] Creating target psm-interop-target-proxy-20230811-2009-igj3k proxy "GRPC" to URL map psm-interop-url-map-20230811-2009-igj3k
I0811 20:09:52.971758 140336156241920 compute.py:530] Creating compute resource:
---
name: psm-interop-target-proxy-20230811-2009-igj3k
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230811-2009-igj3k
validate_for_proxyless: true
...

I0811 20:09:53.510035 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784592973-602ab4bb35ffb-8456ed7b-cd2fc8fb
I0811 20:09:55.958085 140336156241920 traffic_director.py:650] Creating forwarding rule "psm-interop-forwarding-rule-20230811-2009-igj3k" in network "default-vpc": 0.0.0.0:39572 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-2009-igj3k
I0811 20:09:55.960760 140336156241920 compute.py:530] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230811-2009-igj3k
network: global/networks/default-vpc
portRange: 39572
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230811-2009-igj3k
...

I0811 20:09:56.983466 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784595962-602ab4be0fc30-414b9b40-1e19c7a3
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0811 20:10:10.176717 140336156241920 bootstrap_generator_test.py:196] ----- Testing bootstrap generator version v0.14.0 -----
I0811 20:10:10.300733 140336156241920 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-2009-igj3k: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 20:10:10.301132 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-2009-igj3k%22;timeRange=2023-08-11T20:10:10.300974Z%2F2023-08-11T20:40:10.300974Z?project=grpc-testing
I0811 20:10:10.302992 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-2009-igj3k
labels:
name: psm-interop-server-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:10:10.439584 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-2009-igj3k created
I0811 20:10:10.442491 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-2009-igj3k-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 20:10:10.513329 140336156241920 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 20:10:10.513533 140336156241920 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 20:10:10.605893 140336156241920 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zones=['us-central1-a']
I0811 20:10:10.606097 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:10:11.428211 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:10:11.488328 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 20:10:11.495827 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bmuo4dj0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bmuo4dj0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-bmuo4dj0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-2009-igj3k-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:10:11.569507 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 20:10:11.569754 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 20:10:11.676120 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-76c7d5cdd7-5wx8j']
I0811 20:10:11.676426 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-76c7d5cdd7-5wx8j to start
I0811 20:10:14.912837 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-server-76c7d5cdd7-5wx8j ready, IP: 10.120.10.9
I0811 20:10:14.913125 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-76c7d5cdd7-5wx8j to bootstrap_generator_test/test_app_logs/psm-interop-server-20230811-2009-igj3k_psm-grpc-server-76c7d5cdd7-5wx8j.log
I0811 20:10:14.913499 140336065193536 k8s_log_collector.py:70] Starting log collection thread 140336065193536 for psm-grpc-server-76c7d5cdd7-5wx8j
I0811 20:10:14.913635 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 20:10:22.291142 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 20:10:22.328195 140336156241920 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:10:22.622918 140336156241920 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:10:22.623172 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-2009-igj3k-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server', zone='us-central1-a')}
I0811 20:10:22.627187 140336156241920 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 20:10:23.746596 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784622629-602ab4d77e24c-75326885-878ad3ac
I0811 20:11:14.159581 140336156241920 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-2009-igj3k to report all backends healthy: ['psm-interop-server-20230811-2009-igj3k-psm-grpc-server']
I0811 20:11:22.749056 140336156241920 compute.py:487] Backend psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zone us-central1-a reported healthy
I0811 20:11:26.755688 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2009-igj3k: server_target=xds:///psm-grpc-server:39572 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:11:26.756100 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2009-igj3k%22;timeRange=2023-08-11T20:11:26.755931Z%2F2023-08-11T20:41:26.755931Z?project=grpc-testing
I0811 20:11:26.758151 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2009-igj3k
labels:
name: psm-interop-client-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:11:26.836741 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2009-igj3k created
I0811 20:11:26.836997 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:11:27.660668 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:11:27.722418 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:11:27.730082 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-cwhu4idw
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-cwhu4idw
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-cwhu4idw
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:39572"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:11:27.801056 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:11:27.801306 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:11:27.909204 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5c8db4b97d-dxnk9']
I0811 20:11:27.909483 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-5c8db4b97d-dxnk9 to start
I0811 20:11:31.139894 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-5c8db4b97d-dxnk9 ready, IP: 10.120.9.234
I0811 20:11:31.140171 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-5c8db4b97d-dxnk9 to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2009-igj3k_psm-grpc-client-5c8db4b97d-dxnk9.log
I0811 20:11:31.140546 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-5c8db4b97d-dxnk9
I0811 20:11:31.140722 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:11:32.351719 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:11:32.352318 140336156241920 client_app.py:200] [psm-grpc-client-5c8db4b97d-dxnk9] Waiting to report a READY channel to xds:///psm-grpc-server:39572
I0811 20:11:32.450960 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:11:42.495952 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:11:52.540523 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:12:02.585978 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:12:12.630521 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:12:28.680223 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:12:53.739739 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:13:18.799976 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:13:43.859522 140336156241920 client_app.py:236] [psm-grpc-client-5c8db4b97d-dxnk9] Server channel:
I0811 20:13:43.892398 140336156241920 client_app.py:249] [psm-grpc-client-5c8db4b97d-dxnk9] Found subchannel in state READY:
I0811 20:13:43.892725 140336156241920 client_app.py:211] [psm-grpc-client-5c8db4b97d-dxnk9] Channel to xds:///psm-grpc-server:39572 transitioned to state READY:
I0811 20:13:43.893042 140336156241920 grpc.py:74] [psm-grpc-client-5c8db4b97d-dxnk9:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:13:43.927377 140336156241920 grpc.py:74] [psm-grpc-client-5c8db4b97d-dxnk9:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:13:47.926223 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-5c8db4b97d-dxnk9] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76c7d5cdd7-5wx8j: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-76c7d5cdd7-5wx8j: 100

I0811 20:13:47.926527 140336156241920 bootstrap_generator_test.py:172] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0811 20:13:47.929228 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:13:58.186986 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:13:58.824593 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:13:58.943707 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2009-igj3k
I0811 20:14:29.189758 140336156241920 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:14:29.489871 140336156241920 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:14:29.490108 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: set()
I0811 20:14:29.493584 140336156241920 compute.py:537] Patching compute resource:
---
backends: []
...

I0811 20:14:30.283535 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691784869496-602ab5c2ec57f-7618860b-d53fdb5c
I0811 20:15:05.220393 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 20:15:15.399455 140336156241920 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 20:15:15.495867 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:15:16.289193 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 20:15:16.411142 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-2009-igj3k
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0811 20:17:46.906611 140336156241920 bootstrap_generator_test.py:196] ----- Testing bootstrap generator version v0.13.0 -----
I0811 20:17:46.908641 140336156241920 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-2009-igj3k: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 20:17:46.908912 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-2009-igj3k%22;timeRange=2023-08-11T20:17:46.908789Z%2F2023-08-11T20:47:46.908789Z?project=grpc-testing
I0811 20:17:46.910763 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-2009-igj3k
labels:
name: psm-interop-server-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:17:46.970976 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-2009-igj3k created
I0811 20:17:46.973893 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-2009-igj3k-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 20:17:47.046572 140336156241920 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 20:17:47.046802 140336156241920 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 20:17:47.140430 140336156241920 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zones=['us-central1-a']
I0811 20:17:47.140666 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:17:47.863621 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:17:47.925226 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 20:17:47.932519 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4q89yopk
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4q89yopk
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4q89yopk
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230811-2009-igj3k-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:17:48.004903 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 20:17:48.005260 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 20:17:48.108799 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6845d6674d-mx9zj']
I0811 20:17:48.109126 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-6845d6674d-mx9zj to start
I0811 20:17:51.350197 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-server-6845d6674d-mx9zj ready, IP: 10.120.11.245
I0811 20:17:51.350493 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-6845d6674d-mx9zj to bootstrap_generator_test/test_app_logs/psm-interop-server-20230811-2009-igj3k_psm-grpc-server-6845d6674d-mx9zj.log
I0811 20:17:51.350869 140336065193536 k8s_log_collector.py:70] Starting log collection thread 140336065193536 for psm-grpc-server-6845d6674d-mx9zj
I0811 20:17:51.351068 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 20:17:57.686047 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 20:17:57.722318 140336156241920 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:17:57.831889 140336156241920 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:17:57.832117 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-2009-igj3k-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server', zone='us-central1-a')}
I0811 20:17:57.836185 140336156241920 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 20:17:58.729948 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691785077838-602ab6899d263-1b94f553-12df52b3
I0811 20:18:55.814829 140336156241920 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-2009-igj3k to report all backends healthy: ['psm-interop-server-20230811-2009-igj3k-psm-grpc-server']
I0811 20:19:08.883745 140336156241920 compute.py:487] Backend psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zone us-central1-a reported healthy
I0811 20:19:12.890668 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2009-igj3k: server_target=xds:///psm-grpc-server:39572 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:19:12.891027 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2009-igj3k%22;timeRange=2023-08-11T20:19:12.890899Z%2F2023-08-11T20:49:12.890899Z?project=grpc-testing
I0811 20:19:12.892848 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2009-igj3k
labels:
name: psm-interop-client-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:19:12.969517 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2009-igj3k created
I0811 20:19:12.969747 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:19:13.707864 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:19:13.780806 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:19:13.788810 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-baw2urdo
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-baw2urdo
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-baw2urdo
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:39572"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:19:13.857542 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:19:13.857782 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:19:13.965327 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7d5fb7df45-v9kmh']
I0811 20:19:13.965622 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-7d5fb7df45-v9kmh to start
I0811 20:19:17.198786 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-7d5fb7df45-v9kmh ready, IP: 10.120.10.12
I0811 20:19:17.199040 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-7d5fb7df45-v9kmh to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2009-igj3k_psm-grpc-client-7d5fb7df45-v9kmh.log
I0811 20:19:17.199362 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-7d5fb7df45-v9kmh
I0811 20:19:17.199496 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:19:18.340502 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:19:18.341138 140336156241920 client_app.py:200] [psm-grpc-client-7d5fb7df45-v9kmh] Waiting to report a READY channel to xds:///psm-grpc-server:39572
I0811 20:19:18.440050 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:19:28.485281 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:19:38.530182 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:19:48.574839 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:19:58.619678 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:20:14.668743 140336156241920 client_app.py:236] [psm-grpc-client-7d5fb7df45-v9kmh] Server channel:
I0811 20:20:14.700965 140336156241920 client_app.py:249] [psm-grpc-client-7d5fb7df45-v9kmh] Found subchannel in state READY:
I0811 20:20:14.701271 140336156241920 client_app.py:211] [psm-grpc-client-7d5fb7df45-v9kmh] Channel to xds:///psm-grpc-server:39572 transitioned to state READY:
I0811 20:20:14.701540 140336156241920 grpc.py:74] [psm-grpc-client-7d5fb7df45-v9kmh:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:20:14.735403 140336156241920 grpc.py:74] [psm-grpc-client-7d5fb7df45-v9kmh:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:20:18.734521 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-7d5fb7df45-v9kmh] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6845d6674d-mx9zj: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6845d6674d-mx9zj: 100

I0811 20:20:18.734826 140336156241920 bootstrap_generator_test.py:172] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0811 20:20:18.738079 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
I0811 20:20:28.888245 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:20:29.524060 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:20:29.640088 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2009-igj3k
I0811 20:20:59.881517 140336156241920 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:21:00.156358 140336156241920 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:21:00.156601 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: set()
I0811 20:21:00.160075 140336156241920 compute.py:537] Patching compute resource:
---
backends: []
...

I0811 20:21:01.142060 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691785260162-602ab7377ddae-d1e3202f-e3d99ce1
I0811 20:21:52.142262 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 20:22:02.324932 140336156241920 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 20:22:02.444013 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:22:03.045651 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 20:22:03.160985 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-2009-igj3k
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0811 20:24:03.588623 140336156241920 bootstrap_generator_test.py:196] ----- Testing bootstrap generator version v0.12.0 -----
I0811 20:24:03.590728 140336156241920 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-2009-igj3k: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 20:24:03.591025 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-2009-igj3k%22;timeRange=2023-08-11T20:24:03.590904Z%2F2023-08-11T20:54:03.590904Z?project=grpc-testing
I0811 20:24:03.592874 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-2009-igj3k
labels:
name: psm-interop-server-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:24:03.658640 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-2009-igj3k created
I0811 20:24:03.661566 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-2009-igj3k-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 20:24:03.726740 140336156241920 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 20:24:03.726961 140336156241920 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 20:24:03.823185 140336156241920 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zones=['us-central1-a']
I0811 20:24:03.823408 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:24:04.485667 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:24:04.555674 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 20:24:04.563381 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-6s8uigiw
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-6s8uigiw
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-6s8uigiw
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20230811-2009-igj3k-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:24:04.632184 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 20:24:04.632429 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 20:24:04.750193 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-fcf97697b-psv8t']
I0811 20:24:04.750465 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-fcf97697b-psv8t to start
I0811 20:24:07.993146 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-server-fcf97697b-psv8t ready, IP: 10.120.13.4
I0811 20:24:07.993455 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-fcf97697b-psv8t to bootstrap_generator_test/test_app_logs/psm-interop-server-20230811-2009-igj3k_psm-grpc-server-fcf97697b-psv8t.log
I0811 20:24:07.993836 140336065193536 k8s_log_collector.py:70] Starting log collection thread 140336065193536 for psm-grpc-server-fcf97697b-psv8t
I0811 20:24:07.993998 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 20:24:14.408804 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 20:24:14.446061 140336156241920 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:24:14.719524 140336156241920 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:24:14.719742 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-2009-igj3k-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server', zone='us-central1-a')}
I0811 20:24:14.723711 140336156241920 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 20:24:15.785526 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691785454726-602ab7f10ac1a-f1da4d5e-26f9886c
I0811 20:25:08.267277 140336156241920 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-2009-igj3k to report all backends healthy: ['psm-interop-server-20230811-2009-igj3k-psm-grpc-server']
I0811 20:25:08.469918 140336156241920 compute.py:487] Backend psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zone us-central1-a reported healthy
I0811 20:25:12.476821 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2009-igj3k: server_target=xds:///psm-grpc-server:39572 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:25:12.477199 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2009-igj3k%22;timeRange=2023-08-11T20:25:12.477056Z%2F2023-08-11T20:55:12.477056Z?project=grpc-testing
I0811 20:25:12.479292 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2009-igj3k
labels:
name: psm-interop-client-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:25:12.521439 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2009-igj3k created
I0811 20:25:12.521687 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:25:13.188812 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:25:13.248783 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:25:13.256598 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vnsdoqae
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vnsdoqae
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vnsdoqae
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:39572"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:25:13.334012 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:25:13.334285 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:25:13.443587 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-78cbcc5cbd-65xs4']
I0811 20:25:13.443850 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-78cbcc5cbd-65xs4 to start
I0811 20:25:16.677653 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-78cbcc5cbd-65xs4 ready, IP: 10.120.9.238
I0811 20:25:16.677920 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-78cbcc5cbd-65xs4 to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2009-igj3k_psm-grpc-client-78cbcc5cbd-65xs4.log
I0811 20:25:16.678327 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-78cbcc5cbd-65xs4
I0811 20:25:16.678465 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:25:17.821760 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:25:17.822356 140336156241920 client_app.py:200] [psm-grpc-client-78cbcc5cbd-65xs4] Waiting to report a READY channel to xds:///psm-grpc-server:39572
I0811 20:25:17.921131 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:25:27.966376 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:25:38.010860 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:25:48.055487 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:25:58.096826 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:26:14.145817 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:26:39.205243 140336156241920 client_app.py:236] [psm-grpc-client-78cbcc5cbd-65xs4] Server channel:
I0811 20:26:39.237438 140336156241920 client_app.py:249] [psm-grpc-client-78cbcc5cbd-65xs4] Found subchannel in state READY:
I0811 20:26:39.237680 140336156241920 client_app.py:211] [psm-grpc-client-78cbcc5cbd-65xs4] Channel to xds:///psm-grpc-server:39572 transitioned to state READY:
I0811 20:26:39.237888 140336156241920 grpc.py:74] [psm-grpc-client-78cbcc5cbd-65xs4:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:26:39.271755 140336156241920 grpc.py:74] [psm-grpc-client-78cbcc5cbd-65xs4:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:26:43.300506 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-78cbcc5cbd-65xs4] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-fcf97697b-psv8t: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-fcf97697b-psv8t: 100

I0811 20:26:43.300814 140336156241920 bootstrap_generator_test.py:172] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0811 20:26:43.304992 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
I0811 20:26:53.496564 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:26:54.216265 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0811 20:26:54.329990 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2009-igj3k
I0811 20:27:24.572655 140336156241920 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:27:24.853028 140336156241920 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:27:24.853247 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: set()
I0811 20:27:24.856615 140336156241920 compute.py:537] Patching compute resource:
---
backends: []
...

I0811 20:27:25.555784 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691785644860-602ab8a65e3e3-5919c370-50e47aff
I0811 20:28:07.380558 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0811 20:28:17.552881 140336156241920 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0811 20:28:17.652849 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:28:18.199187 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0811 20:28:18.324068 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-2009-igj3k
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0811 20:30:48.864149 140336156241920 bootstrap_generator_test.py:196] ----- Testing bootstrap generator version v0.11.0 -----
I0811 20:30:48.866309 140336156241920 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230811-2009-igj3k: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0811 20:30:48.866606 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230811-2009-igj3k%22;timeRange=2023-08-11T20:30:48.866477Z%2F2023-08-11T21:00:48.866477Z?project=grpc-testing
I0811 20:30:48.868649 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230811-2009-igj3k
labels:
name: psm-interop-server-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:30:48.940849 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-server-20230811-2009-igj3k created
I0811 20:30:48.943763 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230811-2009-igj3k-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0811 20:30:49.012763 140336156241920 k8s_base_runner.py:280] Service psm-grpc-server created
I0811 20:30:49.013053 140336156241920 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0811 20:30:49.114130 140336156241920 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zones=['us-central1-a']
I0811 20:30:49.114381 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:30:49.819785 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:30:49.879941 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0811 20:30:49.887918 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230811-2009-igj3k
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hv3766r6
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hv3766r6
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-hv3766r6
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20230811-2009-igj3k-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:30:49.966202 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0811 20:30:49.966462 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0811 20:30:50.075870 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5f89c74c9b-f42nb']
I0811 20:30:50.076151 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5f89c74c9b-f42nb to start
I0811 20:30:53.313552 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-server-5f89c74c9b-f42nb ready, IP: 10.120.10.15
I0811 20:30:53.313843 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5f89c74c9b-f42nb to bootstrap_generator_test/test_app_logs/psm-interop-server-20230811-2009-igj3k_psm-grpc-server-5f89c74c9b-f42nb.log
I0811 20:30:53.314231 140336065193536 k8s_log_collector.py:70] Starting log collection thread 140336065193536 for psm-grpc-server-5f89c74c9b-f42nb
I0811 20:30:53.314379 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0811 20:30:58.626627 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0811 20:30:58.663612 140336156241920 traffic_director.py:252] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:30:58.799413 140336156241920 traffic_director.py:255] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:30:58.799652 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230811-2009-igj3k-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server', zone='us-central1-a')}
I0811 20:30:58.803627 140336156241920 compute.py:537] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230811-2009-igj3k-psm-grpc-server
maxRatePerEndpoint: 5
...

I0811 20:31:00.011182 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691785858805-602ab97266f91-19d1a536-e9d33d61
I0811 20:31:41.624140 140336156241920 traffic_director.py:290] Waiting for Backend Service psm-interop-backend-service-20230811-2009-igj3k to report all backends healthy: ['psm-interop-server-20230811-2009-igj3k-psm-grpc-server']
I0811 20:31:58.676383 140336156241920 compute.py:487] Backend psm-interop-server-20230811-2009-igj3k-psm-grpc-server in zone us-central1-a reported healthy
I0811 20:32:02.683031 140336156241920 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230811-2009-igj3k: server_target=xds:///psm-grpc-server:39572 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0811 20:32:02.683438 140336156241920 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230811-2009-igj3k%22;timeRange=2023-08-11T20:32:02.683301Z%2F2023-08-11T21:02:02.683301Z?project=grpc-testing
I0811 20:32:02.685366 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230811-2009-igj3k
labels:
name: psm-interop-client-20230811-2009-igj3k
owner: xds-k8s-interop-test
...

I0811 20:32:02.728097 140336156241920 k8s_base_runner.py:280] Namespace psm-interop-client-20230811-2009-igj3k created
I0811 20:32:02.728379 140336156241920 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:32:03.455167 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0811 20:32:03.518096 140336156241920 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0811 20:32:03.526614 140336156241920 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230811-2009-igj3k
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-13zx2owz
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-13zx2owz
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-13zx2owz
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:0af3cad6ea5703485ec781cf0a15b4ae412010a4
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:39572"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0811 20:32:03.600331 140336156241920 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0811 20:32:03.600594 140336156241920 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0811 20:32:03.710572 140336156241920 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5c9558687-dj2sg']
I0811 20:32:03.710901 140336156241920 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-5c9558687-dj2sg to start
I0811 20:32:06.936705 140336156241920 k8s_base_runner.py:554] Pod psm-grpc-client-5c9558687-dj2sg ready, IP: 10.120.13.6
I0811 20:32:06.936993 140336156241920 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-5c9558687-dj2sg to bootstrap_generator_test/test_app_logs/psm-interop-client-20230811-2009-igj3k_psm-grpc-client-5c9558687-dj2sg.log
I0811 20:32:06.937372 140336056800832 k8s_log_collector.py:70] Starting log collection thread 140336056800832 for psm-grpc-client-5c9558687-dj2sg
I0811 20:32:06.937509 140336156241920 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0811 20:32:08.079117 140336156241920 k8s_base_runner.py:522] Deployment psm-grpc-client has 1 replicas available
I0811 20:32:08.079678 140336156241920 client_app.py:200] [psm-grpc-client-5c9558687-dj2sg] Waiting to report a READY channel to xds:///psm-grpc-server:39572
I0811 20:32:08.179748 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:32:18.219420 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:32:28.263991 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:32:38.307957 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:32:48.352655 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:33:04.402009 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:33:29.461835 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:33:54.519055 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:34:19.578893 140336156241920 client_app.py:236] [psm-grpc-client-5c9558687-dj2sg] Server channel:
I0811 20:34:19.611436 140336156241920 client_app.py:249] [psm-grpc-client-5c9558687-dj2sg] Found subchannel in state READY:
I0811 20:34:19.611773 140336156241920 client_app.py:211] [psm-grpc-client-5c9558687-dj2sg] Channel to xds:///psm-grpc-server:39572 transitioned to state READY:
I0811 20:34:19.612033 140336156241920 grpc.py:74] [psm-grpc-client-5c9558687-dj2sg:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0811 20:34:19.646675 140336156241920 grpc.py:74] [psm-grpc-client-5c9558687-dj2sg:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0811 20:34:23.643471 140336156241920 xds_k8s_testcase.py:558] [psm-grpc-client-5c9558687-dj2sg] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5f89c74c9b-f42nb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-5f89c74c9b-f42nb: 100

I0811 20:34:23.643829 140336156241920 bootstrap_generator_test.py:172] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0811 20:34:23.647125 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:34:33.940283 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230811-2009-igj3k/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:34:34.676667 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-client
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:34:34.814155 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230811-2009-igj3k
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:35:05.085045 140336156241920 traffic_director.py:261] Waiting for Network Endpoint Groups to load endpoints.
I0811 20:35:05.357845 140336156241920 traffic_director.py:264] Loaded NEG "psm-interop-server-20230811-2009-igj3k-psm-grpc-server" in zone us-central1-a
I0811 20:35:05.358081 140336156241920 traffic_director.py:273] Adding backends to Backend Service psm-interop-backend-service-20230811-2009-igj3k: set()
I0811 20:35:05.361607 140336156241920 compute.py:537] Patching compute resource:
---
backends: []
...

I0811 20:35:06.413798 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786105364-602aba5d8a040-2969c985-63e91c89
I0811 20:36:05.437858 140336156241920 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:36:15.637953 140336156241920 k8s_base_runner.py:474] Deleting service psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:36:15.744085 140336156241920 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230811-2009-igj3k/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0811 20:36:16.432709 140336156241920 k8s_base_runner.py:487] Deleting service account psm-grpc-server
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0811 20:36:16.560217 140336156241920 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230811-2009-igj3k
/tmp/tmp.VtPSjI0WCh/grpc/tools/run_tests/xds_k8s_test_driver/venv/lib/python3.10/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0811 20:37:16.863102 140336156241920 traffic_director.py:670] Deleting Forwarding rule "psm-interop-forwarding-rule-20230811-2009-igj3k"
I0811 20:37:17.897171 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786236866-602abadaf30fd-e8461463-bb615c5e
I0811 20:37:31.110781 140336156241920 traffic_director.py:709] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230811-2009-igj3k"
I0811 20:37:31.279152 140336156241920 compute.py:561] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:31.279362 140336156241920 traffic_director.py:601] Deleting HTTP Target proxy "psm-interop-target-proxy-20230811-2009-igj3k"
I0811 20:37:31.709288 140336156241920 compute.py:561] Resource targetHttpProxy "psm-interop-target-proxy-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:31.709526 140336156241920 traffic_director.py:589] Deleting Target GRPC proxy "psm-interop-target-proxy-20230811-2009-igj3k"
I0811 20:37:32.222140 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786251712-602abae91b7fd-58c647e2-ff25cb9a
I0811 20:37:34.688735 140336156241920 traffic_director.py:629] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230811-2009-igj3k"
I0811 20:37:35.105917 140336156241920 compute.py:561] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:35.106133 140336156241920 traffic_director.py:521] Deleting URL Map "psm-interop-url-map-20230811-2009-igj3k"
I0811 20:37:36.041183 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786255111-602abaec595d4-0033b5e6-775df0e6
I0811 20:37:38.459003 140336156241920 traffic_director.py:557] Deleting alternative URL Map "psm-interop-url-map-alt-20230811-2009-igj3k"
I0811 20:37:38.817166 140336156241920 compute.py:561] Resource urlMap "psm-interop-url-map-alt-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:38.817418 140336156241920 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230811-2009-igj3k"
I0811 20:37:39.525980 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786258822-602abaefe366b-8e7d8688-fce18c90
I0811 20:37:56.976253 140336156241920 traffic_director.py:328] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230811-2009-igj3k"
I0811 20:37:57.376099 140336156241920 compute.py:561] Resource backendService "psm-interop-backend-service-alt-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:57.376313 140336156241920 traffic_director.py:401] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230811-2009-igj3k"
I0811 20:37:57.772752 140336156241920 compute.py:561] Resource backendService "psm-interop-backend-service-affinity-20230811-2009-igj3k" not deleted since it does not exist
I0811 20:37:57.773001 140336156241920 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230811-2009-igj3k"
I0811 20:37:58.311965 140336156241920 compute.py:589] Waiting 600 sec for compute operation id: operation-1691786277863-602abb020c144-a1442dfc-31e9410c
----------------------------------------------------------------------
Ran 8 tests in 2278.456s

OK
+ echo 'Failed test suites: 0'
Failed test suites: 0


[ID: 4859929] Command finished after 6822 secs, exit value: 0


Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[13:38:03] Collecting build artifacts from build VM
[13:38:05] Kokoro builder finished