Showing build.log
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[11:34:13 PST] Transferring environment variable script to build VM
[11:34:14 PST] Transferring kokoro_log_reader.py to build VM
[11:34:15 PST] Transferring source code to build VM
[11:34:17 PST] Executing build script on build VM
[ID: 3109391] Executing command via SSH:
export KOKORO_BUILD_NUMBER="697"
export KOKORO_JOB_NAME="grpc/go/master/branch/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-go/test/kokoro/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-go/test/kokoro/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-go
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/zasweq/grpc-go.git
GIT_COMMIT_SHORT,bb7b7332
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 130042 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129987 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129587 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_0%3a459.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../kubectl_1%3a459.0.0-0_amd64.deb ...
Unpacking kubectl (1:459.0.0-0) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../python3-pip-whl_22.0.2+dfsg-1ubuntu0.4_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../python3.10-venv_3.10.12-1~22.04.3_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.3) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Setting up kubectl (1:459.0.0-0) ...
Setting up python3.10-venv (3.10.12-1~22.04.3) ...
Processing triggers for man-db (2.10.2-1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.mnM8lQRITP/psm-interop
Cloning into '/tmp/tmp.mnM8lQRITP/psm-interop'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.mnM8lQRITP/psm-interop/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 1.8 MB/s eta 0:00:00
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 7.0 MB/s eta 0:00:00
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 5.8 MB/s eta 0:00:00
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 4.7 MB/s eta 0:00:00
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 7.0 MB/s eta 0:00:00
Collecting grpcio==1.57.0 (from -r requirements.lock (line 6))
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting grpcio-health-checking==1.57.0 (from -r requirements.lock (line 7))
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting grpcio-tools==1.57.0 (from -r requirements.lock (line 8))
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio-channelz==1.57.0 (from -r requirements.lock (line 9))
Downloading grpcio_channelz-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 10))
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 11))
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 12))
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==23.1 (from -r requirements.lock (line 13))
Downloading packaging-23.1-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 5.0 MB/s eta 0:00:00
Collecting Pygments==2.14.0 (from -r requirements.lock (line 14))
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 10.9 MB/s eta 0:00:00
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 15))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 10.6 MB/s eta 0:00:00
Collecting protobuf==4.24.1 (from -r requirements.lock (line 16))
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting xds-protos==1.58.0rc1 (from -r requirements.lock (line 17))
Downloading xds_protos-1.58.0rc1-py3-none-any.whl.metadata (1.3 kB)
Collecting cachetools==5.3.1 (from -r requirements.lock (line 19))
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.lock (line 20))
Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.2.0 (from -r requirements.lock (line 21))
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting google-api-core==2.11.1 (from -r requirements.lock (line 22))
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0 (from -r requirements.lock (line 23))
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 24))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.60.0 (from -r requirements.lock (line 25))
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 26))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.57.0 (from -r requirements.lock (line 27))
Downloading grpcio_status-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting httplib2==0.22.0 (from -r requirements.lock (line 28))
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 8.0 MB/s eta 0:00:00
Collecting idna==3.4 (from -r requirements.lock (line 29))
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 5.5 MB/s eta 0:00:00
Collecting MarkupSafe==2.1.3 (from -r requirements.lock (line 30))
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 31))
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 9.2 MB/s eta 0:00:00
Collecting proto-plus==1.22.3 (from -r requirements.lock (line 32))
Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.lock (line 33))
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 8.2 MB/s eta 0:00:00
Collecting pyasn1-modules==0.3.0 (from -r requirements.lock (line 34))
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 11.0 MB/s eta 0:00:00
Collecting pyparsing==3.1.1 (from -r requirements.lock (line 35))
Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting requests==2.31.0 (from -r requirements.lock (line 36))
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 37))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 38))
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 39))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.16 (from -r requirements.lock (line 40))
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 4.0 MB/s eta 0:00:00
Collecting websocket-client==1.6.2 (from -r requirements.lock (line 41))
Downloading websocket_client-1.6.2-py3-none-any.whl.metadata (7.5 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.57.0->-r requirements.lock (line 8)) (69.0.3)
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 21.6 MB/s eta 0:00:00
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl (8.6 kB)
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.5 MB/s eta 0:00:00
Downloading grpcio_channelz-1.57.0-py3-none-any.whl (10 kB)
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 30.4 MB/s eta 0:00:00
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.4/311.4 kB 23.4 MB/s eta 0:00:00
Downloading xds_protos-1.58.0rc1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 33.8 MB/s eta 0:00:00
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 13.6 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.8/201.8 kB 17.2 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 11.0 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 12.7 MB/s eta 0:00:00
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 19.0 MB/s eta 0:00:00
Downloading grpcio_status-1.57.0-py3-none-any.whl (5.1 kB)
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 4.1 MB/s eta 0:00:00
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 10.8 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 6.2 MB/s eta 0:00:00
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 13.6 MB/s eta 0:00:00
Downloading websocket_client-1.6.2-py3-none-any.whl (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 5.5 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, uritemplate, six, PyYAML, pyparsing, Pygments, pyasn1, protobuf, packaging, oauthlib, MarkupSafe, idna, grpcio, charset-normalizer, certifi, cachetools, xds-protos, tenacity, rsa, requests, python-dateutil, pyasn1-modules, proto-plus, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, absl-py, requests-oauthlib, grpcio-status, google-auth, kubernetes, grpc-google-iam-v1, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-secret-manager
Successfully installed Mako-1.2.4 MarkupSafe-2.1.3 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-api-core-2.11.1 google-api-python-client-1.12.11 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.57.0 grpcio-channelz-1.57.0 grpcio-health-checking-1.57.0 grpcio-status-1.57.0 grpcio-tools-1.57.0 httplib2-0.22.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.1 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.16 websocket-client-1.6.2 xds-protos-1.58.0rc1
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.1
certifi 2023.7.22
charset-normalizer 3.2.0
google-api-core 2.11.1
google-api-python-client 1.12.11
google-auth 2.22.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.60.0
grpc-google-iam-v1 0.12.6
grpcio 1.57.0
grpcio-channelz 1.57.0
grpcio-health-checking 1.57.0
grpcio-status 1.57.0
grpcio-tools 1.57.0
httplib2 0.22.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.3
oauthlib 3.2.2
packaging 23.1
pip 23.3.2
proto-plus 1.22.3
protobuf 4.24.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
Pygments 2.14.0
pyparsing 3.1.1
python-dateutil 2.8.2
PyYAML 6.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.0.3
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.16
websocket-client 1.6.2
xds-protos 1.58.0rc1
Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto
/tmp/tmp.mnM8lQRITP/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.mnM8lQRITP/psm-interop/protos/grpc/testing:
total 108
-rw-rw-r-- 1 kbuilder kbuilder 957 Jan 11 19:35 empty.proto
-rw-rw-r-- 1 kbuilder kbuilder 1007 Jan 11 19:35 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 11 19:35 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 11527 Jan 11 19:35 messages.proto
-rw-rw-r-- 1 kbuilder kbuilder 17206 Jan 11 19:35 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 11 19:35 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 4871 Jan 11 19:35 test.proto
-rw-rw-r-- 1 kbuilder kbuilder 4296 Jan 11 19:35 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 41870 Jan 11 19:35 test_pb2_grpc.py
Server image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
Server image not found
Client image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
Client image not found
Building Go xDS interop test app Docker images
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.51kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.6s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.6s
#5 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#5 resolve docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 0.0s done
#5 sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 1.64kB / 1.64kB done
#5 sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd 528B / 528B done
#5 sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418 1.47kB / 1.47kB done
#5 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.1s done
#5 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca
#5 DONE 0.5s
#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#6 resolve docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 0.0s done
#6 sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 1.65kB / 1.65kB done
#6 sha256:2523a6f68a0f515fe251aad40b18545155135ca6a5b2e61da8254df9153e3648 1.16kB / 1.16kB done
#6 sha256:82e66f0646744e5c40c91b75ff7d5ff5577887b46d5402965633324dd842ea87 5.22kB / 5.22kB done
#6 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.1s done
#6 sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 284.20kB / 284.20kB 0.1s done
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 29.36MB / 67.09MB 0.5s
#6 sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 156B / 156B 0.2s done
#6 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0.1s done
#6 extracting sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 0.1s done
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 44.04MB / 67.09MB 0.6s
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 57.67MB / 67.09MB 0.7s
#6 ...
#7 [internal] load build context
#7 transferring context: 87.13MB 0.8s done
#7 DONE 0.8s
#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 67.09MB / 67.09MB 1.0s done
#6 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 0.1s
#6 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 4.7s done
#6 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98
#6 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 done
#6 DONE 6.1s
#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 DONE 1.2s
#9 [build 3/4] COPY . .
#9 DONE 0.5s
#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/client/client.go
#10 0.363 go: downloading golang.org/x/net v0.18.0
#10 0.377 go: downloading github.com/golang/protobuf v1.5.3
#10 0.380 go: downloading google.golang.org/protobuf v1.31.0
#10 0.394 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.406 go: downloading golang.org/x/sys v0.14.0
#10 0.439 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.468 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.838 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.845 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.893 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.340 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.340 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.343 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.343 go: downloading golang.org/x/text v0.14.0
#10 1.346 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.346 go: downloading golang.org/x/sync v0.5.0
#10 1.410 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.419 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 19.3s
#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/client .
#11 DONE 0.1s
#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:b5bd887113ca73ccd5b3949a23db4a47a846a45ad140c6f910e259aa76ce9fee done
#12 naming to gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b done
#12 DONE 0.2s
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.51kB done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.2s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.2s
#5 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#5 DONE 0.0s
#6 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#6 CACHED
#7 [internal] load build context
#7 transferring context: 76.35kB 0.1s done
#7 DONE 0.1s
#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 CACHED
#9 [build 3/4] COPY . .
#9 CACHED
#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/server/server.go
#10 0.319 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.334 go: downloading google.golang.org/protobuf v1.31.0
#10 0.335 go: downloading github.com/golang/protobuf v1.5.3
#10 0.336 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.356 go: downloading golang.org/x/sys v0.14.0
#10 0.391 go: downloading golang.org/x/net v0.18.0
#10 0.394 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.755 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.823 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.853 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.313 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.313 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.319 go: downloading golang.org/x/text v0.14.0
#10 1.322 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.326 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.336 go: downloading golang.org/x/sync v0.5.0
#10 1.393 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.406 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 19.4s
#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/server .
#11 DONE 0.1s
#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:ace14729962dac25c084bafd8f0e2175a07a1a563e7384aecb5b447cba7048bf done
#12 naming to gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b done
#12 DONE 0.2s
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-client]
5cf64c40fd39: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
5cf64c40fd39: Pushed
bb7b7332b803301b57f6497415512a9f5471fa0b: digest: sha256:d174949a1da645bbdef7d5ece56b02f3fe5a12476216ef2b817fd18c16d8a960 size: 740
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-server]
071b551139b6: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
071b551139b6: Pushed
bb7b7332b803301b57f6497415512a9f5471fa0b: digest: sha256:db9bd17b6781e905324dba79d831ff007c7932068f2904eae7653ffabd59b5ff size: 740
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test'
+ python -m tests.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:36:34.201105 139848621117440 xds_k8s_testcase.py:157] ----- Testing AffinityTest -----
I0111 19:36:34.201264 139848621117440 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:36:34.201531 139848621117440 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:36:34.218817 139848621117440 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:36:34.375864 139848621117440 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0111 19:36:34.652938 139848621117440 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1936-cyja4
I0111 19:36:34.948297 139848621117440 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test
I0111 19:36:35.343053 139848621117440 xds_k8s_testcase.py:663] Found unused xds port: 41089
I0111 19:36:35.343407 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0111 19:36:35.343583 139848621117440 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1936-cyja4"
I0111 19:36:35.346664 139848621117440 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1936-cyja4
type: GRPC
...
I0111 19:36:36.106127 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001795411-60eb0aa5b5d3c-b1fa0bd1-714988f4
I0111 19:36:38.401229 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0111 19:36:38.401441 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0111 19:36:38.401545 139848621117440 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1936-cyja4"
I0111 19:36:38.409992 139848621117440 compute.py:537] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1936-cyja4
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20240111-1936-cyja4
protocol: GRPC
sessionAffinity: HEADER_FIELD
...
I0111 19:36:39.205381 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001798412-60eb0aa8927ef-483612cf-c38a30e8
I0111 19:37:35.390896 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0111 19:37:35.391151 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0111 19:37:35.391245 139848621117440 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1936-cyja4": psm-grpc-server:41089 -> psm-interop-backend-service-20240111-1936-cyja4
I0111 19:37:35.402735 139848621117440 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4
hostRules:
- hosts:
- psm-grpc-server:41089
pathMatcher: psm-interop-path-matcher-20240111-1936-cyja4
name: psm-interop-url-map-20240111-1936-cyja4
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4
name: psm-interop-path-matcher-20240111-1936-cyja4
...
I0111 19:37:35.989221 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001855405-60eb0adeecbf8-445debde-cdf8b08e
I0111 19:37:38.343500 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0111 19:37:38.343738 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0111 19:37:38.343828 139848621117440 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1936-cyja4 proxy "GRPC" to URL map psm-interop-url-map-20240111-1936-cyja4
I0111 19:37:38.345857 139848621117440 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1936-cyja4
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1936-cyja4
validate_for_proxyless: true
...
I0111 19:37:38.870654 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001858347-60eb0ae1bb383-7a47c5d4-e838477a
I0111 19:37:41.133113 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0111 19:37:41.133332 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0111 19:37:41.133435 139848621117440 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1936-cyja4" in network "default-vpc": 0.0.0.0:41089 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1936-cyja4
I0111 19:37:41.135988 139848621117440 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1936-cyja4
network: global/networks/default-vpc
portRange: 41089
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1936-cyja4
...
I0111 19:37:42.121852 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001861138-60eb0ae4646a8-aac284d3-433fcac3
I0111 19:37:57.333106 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0111 19:37:57.333347 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0111 19:37:57.333483 139848621117440 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1936-cyja4: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0111 19:37:57.333723 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1936-cyja4%22;timeRange=2024-01-11T19:37:57.333568Z%2F2024-01-11T20:07:57.333568Z?project=grpc-testing
I0111 19:37:57.338714 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1936-cyja4
labels:
name: psm-interop-server-20240111-1936-cyja4
owner: xds-k8s-interop-test
...
I0111 19:37:57.419703 139848621117440 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1936-cyja4 created
I0111 19:37:57.422587 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1936-cyja4-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:37:57.485134 139848621117440 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:37:57.485352 139848621117440 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1936-cyja4
I0111 19:37:57.582336 139848621117440 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1936-cyja4: neg_name=psm-interop-server-20240111-1936-cyja4-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:37:57.582550 139848621117440 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1936-cyja4/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:37:58.342348 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:37:58.401875 139848621117440 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:37:58.410441 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1936-cyja4-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:37:58.478842 139848621117440 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:37:58.479103 139848621117440 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0111 19:37:58.634901 139848621117440 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-5cfd6545d6-5hmdg', 'psm-grpc-server-5cfd6545d6-9ckr4', 'psm-grpc-server-5cfd6545d6-bgjgs']
I0111 19:37:58.635306 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-5hmdg to start
I0111 19:38:01.947062 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-5hmdg ready, IP: 10.120.3.99
I0111 19:38:01.947337 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-5hmdg to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-5hmdg.log
I0111 19:38:01.947907 139848528160320 k8s_log_collector.py:70] Starting log collection thread 139848528160320 for psm-grpc-server-5cfd6545d6-5hmdg
I0111 19:38:01.948068 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-9ckr4 to start
I0111 19:38:02.052683 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-9ckr4 ready, IP: 10.120.0.65
I0111 19:38:02.052969 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-9ckr4 to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-9ckr4.log
I0111 19:38:02.053452 139848519767616 k8s_log_collector.py:70] Starting log collection thread 139848519767616 for psm-grpc-server-5cfd6545d6-9ckr4
I0111 19:38:02.053621 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-bgjgs to start
I0111 19:38:02.231600 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-bgjgs ready, IP: 10.120.2.167
I0111 19:38:02.231879 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-bgjgs to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-bgjgs.log
I0111 19:38:02.232368 139848511374912 k8s_log_collector.py:70] Starting log collection thread 139848511374912 for psm-grpc-server-5cfd6545d6-bgjgs
I0111 19:38:02.232539 139848621117440 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0111 19:38:08.634159 139848621117440 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0111 19:38:08.634728 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0111 19:38:08.634852 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0111 19:38:08.670674 139848621117440 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:38:08.949388 139848621117440 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1936-cyja4-psm-grpc-server" in zone us-central1-a
I0111 19:38:08.949620 139848621117440 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1936-cyja4: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1936-cyja4-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1936-cyja4-psm-grpc-server', zone='us-central1-a')}
I0111 19:38:08.953987 139848621117440 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1936-cyja4-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:38:09.961627 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001888956-60eb0afeec086-0cabeb4c-4f57f99f
I0111 19:38:57.624413 139848621117440 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1936-cyja4 to report all backends healthy: ['psm-interop-server-20240111-1936-cyja4-psm-grpc-server']
I0111 19:39:10.691787 139848621117440 compute.py:494] Backend psm-interop-server-20240111-1936-cyja4-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:39:14.696536 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0111 19:39:14.696756 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0111 19:39:14.696866 139848621117440 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1936-cyja4: server_target=xds:///psm-grpc-server:41089 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0111 19:39:14.697126 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1936-cyja4%22;timeRange=2024-01-11T19:39:14.696992Z%2F2024-01-11T20:09:14.696992Z?project=grpc-testing
I0111 19:39:14.698979 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1936-cyja4
labels:
name: psm-interop-client-20240111-1936-cyja4
owner: xds-k8s-interop-test
...
I0111 19:39:14.740170 139848621117440 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1936-cyja4 created
I0111 19:39:14.740427 139848621117440 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1936-cyja4/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:39:15.314923 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:39:15.374793 139848621117440 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:39:15.383351 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1936-cyja4
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:41089"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:39:15.453018 139848621117440 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:39:15.453255 139848621117440 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:39:15.567415 139848621117440 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6c4558687b-m6qdq']
I0111 19:39:15.567699 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6c4558687b-m6qdq to start
I0111 19:39:18.814490 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-client-6c4558687b-m6qdq ready, IP: 10.120.4.247
I0111 19:39:18.814749 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6c4558687b-m6qdq to affinity_test/test_app_logs/psm-interop-client-20240111-1936-cyja4_psm-grpc-client-6c4558687b-m6qdq.log
I0111 19:39:18.815193 139848295904832 k8s_log_collector.py:70] Starting log collection thread 139848295904832 for psm-grpc-client-6c4558687b-m6qdq
I0111 19:39:18.815328 139848621117440 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:39:20.033506 139848621117440 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:39:20.034057 139848621117440 client_app.py:284] [psm-grpc-client-6c4558687b-m6qdq] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:39:20.137614 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:30.183244 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:40.229298 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:50.274333 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:00.319302 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:16.368706 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:18.403790 139848621117440 client_app.py:323] [psm-grpc-client-6c4558687b-m6qdq] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:40:18.404110 139848621117440 client_app.py:294] [psm-grpc-client-6c4558687b-m6qdq] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:40:18.404378 139848621117440 client_app.py:244] [psm-grpc-client-6c4558687b-m6qdq] Waiting to report a READY channel to xds:///psm-grpc-server:41089
I0111 19:40:18.436961 139848621117440 client_app.py:371] [psm-grpc-client-6c4558687b-m6qdq] Server channel:
I0111 19:40:18.534163 139848621117440 client_app.py:384] [psm-grpc-client-6c4558687b-m6qdq] Found subchannel in state READY:
I0111 19:40:18.534593 139848621117440 client_app.py:255] [psm-grpc-client-6c4558687b-m6qdq] Channel to xds:///psm-grpc-server:41089 transitioned to state READY:
I0111 19:40:18.534864 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:18.569118 139848621117440 affinity_test.py:108] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/990b5fc1-3a6e-460f-b02e-864bf8a72be6",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:40:18.037648090Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:41089"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:40:18.037824090Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1936-cyja4"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:40:18.037374940Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "2",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7886369199152299386_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.2.167",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.99",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.65",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:40:12.460373819Z",
"clientStatus": "ACKED"
}
]
}
I0111 19:40:18.569902 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0111 19:40:18.570007 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0111 19:40:18.570116 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:18.603873 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0111 19:40:18.604084 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0111 19:40:18.604303 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:22.628929 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-9ckr4: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-9ckr4: 100
I0111 19:40:22.629190 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0111 19:40:22.629281 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0111 19:40:22.629405 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:26.628989 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-9ckr4: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-9ckr4: 100
I0111 19:40:26.759898 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0111 19:40:26.760120 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0111 19:40:26.760218 139848621117440 server_app.py:98] [psm-grpc-server-5cfd6545d6-9ckr4] >> Setting health status to NOT_SERVING
I0111 19:40:26.760625 139848621117440 grpc.py:75] [psm-grpc-server-5cfd6545d6-9ckr4:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.858044 139848621117440 grpc.py:75] [psm-grpc-server-5cfd6545d6-9ckr4:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.890568 139848621117440 server_app.py:102] [psm-grpc-server-5cfd6545d6-9ckr4] << Health status status: NOT_SERVING
I0111 19:40:26.890804 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0111 19:40:26.890896 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0111 19:40:26.891036 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.924497 139848621117440 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0111 19:40:36.934947 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:36.970628 139848621117440 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0111 19:40:46.981101 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:47.016489 139848621117440 affinity_test.py:185] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/990b5fc1-3a6e-460f-b02e-864bf8a72be6",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:40:37.901593167Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:41089"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:40:37.902302107Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1936-cyja4"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:40:37.901300707Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "6",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7886369199152299386_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.2.167",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.99",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.65",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:40:42.460178970Z",
"clientStatus": "ACKED"
}
]
}
I0111 19:40:47.017225 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0111 19:40:47.017334 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0111 19:40:47.017457 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:51.028639 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-bgjgs: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-bgjgs: 100
I0111 19:40:51.028964 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0111 19:40:51.029119 139848621117440 xds_k8s_testcase.py:687] ----- TestMethod __main__.AffinityTest.test_affinity teardown -----
I0111 19:40:51.280316 139848621117440 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1936-cyja4"
I0111 19:40:51.998042 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002051284-60eb0b99badda-38aac552-bf4f7a65
I0111 19:41:15.804803 139848621117440 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1936-cyja4"
I0111 19:41:15.928511 139848621117440 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1936-cyja4"
I0111 19:41:16.476036 139848621117440 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1936-cyja4"
I0111 19:41:16.920675 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002076480-60eb0bb1c2488-43294e1a-18c74b81
I0111 19:41:19.139220 139848621117440 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1936-cyja4"
I0111 19:41:19.646523 139848621117440 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1936-cyja4"
I0111 19:41:20.403494 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002079652-60eb0bb4c8941-c33aae89-3ef450ca
I0111 19:41:24.818637 139848621117440 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1936-cyja4"
I0111 19:41:25.276229 139848621117440 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1936-cyja4"
I0111 19:41:25.903390 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002085282-60eb0bba270f6-a55f8819-c1cfa9ba
I0111 19:41:58.339715 139848621117440 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1936-cyja4"
I0111 19:41:58.822424 139848621117440 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1936-cyja4"
I0111 19:41:59.310904 139848621117440 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1936-cyja4"
I0111 19:41:59.927478 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002119406-60eb0bdab24a6-c8441f46-1b4e11f9
I0111 19:42:04.437045 139848621117440 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 19:42:14.630590 139848621117440 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 19:42:14.630914 139848621117440 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1936-cyja4/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:42:15.189674 139848621117440 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 19:42:15.305894 139848621117440 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 19:42:15.306201 139848621117440 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1936-cyja4
I0111 19:42:45.491417 139848621117440 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1936-cyja4 deleted
I0111 19:42:45.491656 139848621117440 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 19:42:55.722656 139848621117440 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 19:42:55.722935 139848621117440 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 19:42:55.837761 139848621117440 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 19:42:55.838038 139848621117440 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1936-cyja4/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:42:57.888352 139848621117440 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 19:42:58.003778 139848621117440 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 19:42:58.004035 139848621117440 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1936-cyja4
I0111 19:44:28.326913 139848621117440 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1936-cyja4 deleted
I0111 19:44:28.327356 139848621117440 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 19:44:28.327643 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-yf9833wq:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1936-cyja4%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-yf9833wq%22;timeRange=2024-01-11T19:39:14.696984Z%2F2024-01-11T19:42:45.491625Z?project=grpc-testing
I0111 19:44:28.327806 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-4soyczp8:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1936-cyja4%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-4soyczp8%22;timeRange=2024-01-11T19:37:57.333559Z%2F2024-01-11T19:44:28.327190Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
I0111 19:44:28.328090 139848621117440 base_testcase.py:52] ----- TestCase __main__.AffinityTest.test_affinity PASSED -----
----------------------------------------------------------------------
Ran 1 test in 474.130s
OK
+ for test in "${test_suites[@]}"
+ run_test api_listener_test
+ local test_name=api_listener_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test'
+ set -x
+ python -m tests.api_listener_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:44:29.736641 140453231611904 xds_k8s_testcase.py:157] ----- Testing ApiListenerTest -----
I0111 19:44:29.736788 140453231611904 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:44:29.737011 140453231611904 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:44:29.754577 140453231611904 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:44:29.771473 140453231611904 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0111 19:44:29.776247 140453231611904 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1944-hkz7f
I0111 19:44:30.091493 140453231611904 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
I0111 19:44:30.534574 140453231611904 xds_k8s_testcase.py:663] Found unused xds port: 58545
I0111 19:44:30.534909 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0111 19:44:30.534989 140453231611904 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1944-hkz7f"
I0111 19:44:30.538130 140453231611904 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1944-hkz7f
type: GRPC
...
I0111 19:44:31.270478 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002270610-60eb0c6ae555d-094ac120-66e891ea
I0111 19:44:33.645896 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0111 19:44:33.646111 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0111 19:44:33.646182 140453231611904 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1944-hkz7f"
I0111 19:44:33.654427 140453231611904 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1944-hkz7f
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-1944-hkz7f
protocol: GRPC
...
I0111 19:44:34.509898 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002273656-60eb0c6dccd5b-d016d964-7fbdcb46
I0111 19:44:54.166541 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0111 19:44:54.166707 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0111 19:44:54.166772 140453231611904 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1944-hkz7f": psm-grpc-server:58545 -> psm-interop-backend-service-20240111-1944-hkz7f
I0111 19:44:54.178301 140453231611904 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
hostRules:
- hosts:
- psm-grpc-server:58545
pathMatcher: psm-interop-path-matcher-20240111-1944-hkz7f
name: psm-interop-url-map-20240111-1944-hkz7f
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
name: psm-interop-path-matcher-20240111-1944-hkz7f
...
I0111 19:44:54.784215 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002294180-60eb0c815fb18-12ef2ed4-31d69a12
I0111 19:44:57.060353 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0111 19:44:57.060564 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0111 19:44:57.060652 140453231611904 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1944-hkz7f proxy "GRPC" to URL map psm-interop-url-map-20240111-1944-hkz7f
I0111 19:44:57.062730 140453231611904 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1944-hkz7f
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1944-hkz7f
validate_for_proxyless: true
...
I0111 19:44:57.639843 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002297064-60eb0c841fc4a-45ffd04c-71c75153
I0111 19:44:59.951745 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0111 19:44:59.951989 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0111 19:44:59.952111 140453231611904 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f" in network "default-vpc": 0.0.0.0:58545 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1944-hkz7f
I0111 19:44:59.954758 140453231611904 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1944-hkz7f
network: global/networks/default-vpc
portRange: 58545
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1944-hkz7f
...
I0111 19:45:00.778454 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002299956-60eb0c86e1d81-30a568ab-d89e7374
I0111 19:45:45.941831 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0111 19:45:45.942088 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0111 19:45:45.942193 140453231611904 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1944-hkz7f: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:45:45.942449 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1944-hkz7f%22;timeRange=2024-01-11T19:45:45.942290Z%2F2024-01-11T20:15:45.942290Z?project=grpc-testing
I0111 19:45:45.947467 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1944-hkz7f
labels:
name: psm-interop-server-20240111-1944-hkz7f
owner: xds-k8s-interop-test
...
I0111 19:45:46.095828 140453231611904 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1944-hkz7f created
I0111 19:45:46.098723 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1944-hkz7f-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:45:46.169998 140453231611904 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:45:46.170229 140453231611904 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1944-hkz7f
I0111 19:45:46.263880 140453231611904 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1944-hkz7f: neg_name=psm-interop-server-20240111-1944-hkz7f-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:45:46.264085 140453231611904 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1944-hkz7f/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:45:46.995814 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:45:47.064201 140453231611904 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:45:47.073744 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1944-hkz7f-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:45:47.142991 140453231611904 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:45:47.143227 140453231611904 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 19:45:47.246189 140453231611904 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6d889f745f-w8d94']
I0111 19:45:47.246444 140453231611904 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6d889f745f-w8d94 to start
I0111 19:45:50.484474 140453231611904 k8s_base_runner.py:820] Pod psm-grpc-server-6d889f745f-w8d94 ready, IP: 10.120.0.66
I0111 19:45:50.484750 140453231611904 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6d889f745f-w8d94 to api_listener_test/test_app_logs/psm-interop-server-20240111-1944-hkz7f_psm-grpc-server-6d889f745f-w8d94.log
I0111 19:45:50.485298 140453138130496 k8s_log_collector.py:70] Starting log collection thread 140453138130496 for psm-grpc-server-6d889f745f-w8d94
I0111 19:45:50.485541 140453231611904 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 19:45:56.837117 140453231611904 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 19:45:56.837530 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0111 19:45:56.837635 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0111 19:45:56.873633 140453231611904 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:45:57.189773 140453231611904 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1944-hkz7f-psm-grpc-server" in zone us-central1-a
I0111 19:45:57.190027 140453231611904 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1944-hkz7f: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1944-hkz7f-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1944-hkz7f-psm-grpc-server', zone='us-central1-a')}
I0111 19:45:57.194467 140453231611904 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1944-hkz7f-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:45:58.067478 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002357197-60eb0cbd78beb-d2177fa3-3bc3c373
I0111 19:46:54.211319 140453231611904 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1944-hkz7f to report all backends healthy: ['psm-interop-server-20240111-1944-hkz7f-psm-grpc-server']
I0111 19:47:07.208019 140453231611904 compute.py:494] Backend psm-interop-server-20240111-1944-hkz7f-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:47:11.212684 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0111 19:47:11.212927 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0111 19:47:11.213034 140453231611904 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1944-hkz7f: server_target=xds:///psm-grpc-server:58545 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 19:47:11.213273 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1944-hkz7f%22;timeRange=2024-01-11T19:47:11.213132Z%2F2024-01-11T20:17:11.213132Z?project=grpc-testing
I0111 19:47:11.215191 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1944-hkz7f
labels:
name: psm-interop-client-20240111-1944-hkz7f
owner: xds-k8s-interop-test
...
I0111 19:47:11.257156 140453231611904 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1944-hkz7f created
I0111 19:47:11.257391 140453231611904 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1944-hkz7f/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:47:11.863847 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:47:11.930180 140453231611904 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:47:12.003937 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1944-hkz7f
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:58545"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:47:12.076130 140453231611904 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:47:12.076387 140453231611904 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:47:12.180996 140453231611904 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-fcdb59d46-8vr7t']
I0111 19:47:12.181296 140453231611904 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-fcdb59d46-8vr7t to start
I0111 19:47:15.418415 140453231611904 k8s_base_runner.py:820] Pod psm-grpc-client-fcdb59d46-8vr7t ready, IP: 10.120.4.248
I0111 19:47:15.418705 140453231611904 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-fcdb59d46-8vr7t to api_listener_test/test_app_logs/psm-interop-client-20240111-1944-hkz7f_psm-grpc-client-fcdb59d46-8vr7t.log
I0111 19:47:15.419272 140453129737792 k8s_log_collector.py:70] Starting log collection thread 140453129737792 for psm-grpc-client-fcdb59d46-8vr7t
I0111 19:47:15.419488 140453231611904 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:47:16.560678 140453231611904 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:47:16.561240 140453231611904 client_app.py:284] [psm-grpc-client-fcdb59d46-8vr7t] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:47:16.669472 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:26.716017 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:36.761762 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:46.807155 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:56.852962 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:12.902960 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:37.963985 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:39.999708 140453231611904 client_app.py:323] [psm-grpc-client-fcdb59d46-8vr7t] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:48:39.999959 140453231611904 client_app.py:294] [psm-grpc-client-fcdb59d46-8vr7t] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:48:40.000201 140453231611904 client_app.py:244] [psm-grpc-client-fcdb59d46-8vr7t] Waiting to report a READY channel to xds:///psm-grpc-server:58545
I0111 19:48:40.033413 140453231611904 client_app.py:371] [psm-grpc-client-fcdb59d46-8vr7t] Server channel:
I0111 19:48:40.066776 140453231611904 client_app.py:384] [psm-grpc-client-fcdb59d46-8vr7t] Found subchannel in state READY:
I0111 19:48:40.067118 140453231611904 client_app.py:255] [psm-grpc-client-fcdb59d46-8vr7t] Channel to xds:///psm-grpc-server:58545 transitioned to state READY:
I0111 19:48:40.067219 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0111 19:48:40.067281 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0111 19:48:40.067470 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:48:40.103037 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0111 19:48:40.103269 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0111 19:48:40.103491 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:48:44.106116 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:48:44.106467 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0111 19:48:44.106555 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0111 19:48:44.106630 140453231611904 traffic_director.py:540] Creating alternative URL map "psm-interop-url-map-alt-20240111-1944-hkz7f": psm-grpc-server:58545 -> psm-interop-backend-service-20240111-1944-hkz7f
I0111 19:48:44.111719 140453231611904 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
hostRules:
- hosts:
- psm-grpc-server:58545
pathMatcher: psm-interop-path-matcher-20240111-1944-hkz7f
name: psm-interop-url-map-alt-20240111-1944-hkz7f
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
name: psm-interop-path-matcher-20240111-1944-hkz7f
...
I0111 19:48:44.827340 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002524115-60eb0d5ca80cf-223e9215-6b043585
I0111 19:48:47.093788 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0111 19:48:47.094034 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0111 19:48:47.094131 140453231611904 traffic_director.py:613] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20240111-1944-hkz7f" to URL map psm-interop-url-map-alt-20240111-1944-hkz7f
I0111 19:48:47.096434 140453231611904 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20240111-1944-hkz7f
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20240111-1944-hkz7f
validate_for_proxyless: false
...
I0111 19:48:47.748824 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002527098-60eb0d5f805fd-db12f20a-ff89db66
I0111 19:48:50.037725 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0111 19:48:50.037978 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0111 19:48:50.038092 140453231611904 traffic_director.py:683] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20240111-1944-hkz7f" in network "default-vpc": 10.10.10.10:58545 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240111-1944-hkz7f
I0111 19:48:50.040755 140453231611904 compute.py:537] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20240111-1944-hkz7f
network: global/networks/default-vpc
portRange: 58545
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240111-1944-hkz7f
...
I0111 19:48:50.916442 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002530043-60eb0d624f53b-ddea5fe5-7646d95d
I0111 19:49:29.771052 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0111 19:49:29.771300 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0111 19:49:29.771491 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:49:33.785936 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:49:33.786256 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:49:33.820180 140453231611904 api_listener_test.py:110] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/2381cab1-578e-4fc1-89a4-1f961af362ab",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.248",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"versionInfo": "8",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_6346379817286237061_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.66",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:49:31.317665801Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:58545",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:58545",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:49:30.867656445Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:58545"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:49:30.867983355Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1944-hkz7f"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:49:30.867038635Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1705002563355708307
I0111 19:49:33.820889 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0111 19:49:33.821023 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0111 19:49:33.821107 140453231611904 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f"
I0111 19:49:34.477007 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002573824-60eb0d8c101d3-df182fdf-f9bca504
I0111 19:49:47.543639 140453231611904 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:49:47.987516 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002587546-60eb0d9926473-a3f9b8c2-5b3c5804
I0111 19:49:50.305828 140453231611904 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1944-hkz7f"
I0111 19:49:51.012564 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002590311-60eb0d9bc93a6-f50a21a0-630c58d6
I0111 19:49:53.262507 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0111 19:49:53.262724 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0111 19:49:53.263120 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:49:57.266071 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:49:57.266360 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:49:57.300235 140453231611904 xds_k8s_testcase.py:552] Routing config not propagated yet. Retrying.
I0111 19:49:57.300778 140453231611904 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0111 19:49:59.303178 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:03.306487 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:03.306796 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:50:03.340737 140453231611904 xds_k8s_testcase.py:552] Routing config not propagated yet. Retrying.
I0111 19:50:03.341084 140453231611904 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0111 19:50:05.343477 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:09.346052 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:09.346329 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:50:09.380117 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:13.386589 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:13.386839 140453231611904 xds_k8s_testcase.py:561] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1705002585732599240
I0111 19:50:13.387053 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0111 19:50:13.387200 140453231611904 xds_k8s_testcase.py:687] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0111 19:50:13.610906 140453231611904 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f"
I0111 19:50:13.740448 140453231611904 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1944-hkz7f"
I0111 19:50:14.356550 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002613743-60eb0db221fba-56ec9e65-c03feb12
I0111 19:50:27.317075 140453231611904 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:50:27.835503 140453231611904 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:50:28.257371 140453231611904 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1944-hkz7f"
I0111 19:50:28.658040 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002628260-60eb0dbffa341-681f5ec5-8358f919
I0111 19:50:30.900532 140453231611904 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1944-hkz7f"
I0111 19:50:31.314488 140453231611904 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1944-hkz7f"
I0111 19:50:32.053630 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002631320-60eb0dc2e546d-89822ca0-37ace6c1
I0111 19:50:34.395989 140453231611904 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1944-hkz7f"
I0111 19:50:35.074580 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002634401-60eb0dc5d574d-ac61f050-147c35b2
I0111 19:50:58.845590 140453231611904 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1944-hkz7f"
I0111 19:50:59.289386 140453231611904 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1944-hkz7f"
I0111 19:50:59.701323 140453231611904 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1944-hkz7f"
I0111 19:51:00.204261 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002659779-60eb0dde0930e-859f9209-3c0940dd
I0111 19:51:04.714053 140453231611904 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 19:51:14.894882 140453231611904 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 19:51:14.895221 140453231611904 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1944-hkz7f/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:51:15.497268 140453231611904 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 19:51:15.619240 140453231611904 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 19:51:15.619474 140453231611904 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1944-hkz7f
I0111 19:51:45.805962 140453231611904 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1944-hkz7f deleted
I0111 19:51:45.806253 140453231611904 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 19:51:55.980636 140453231611904 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 19:51:55.980950 140453231611904 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 19:51:56.097112 140453231611904 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 19:51:56.097378 140453231611904 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1944-hkz7f/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:51:56.825804 140453231611904 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 19:51:56.940458 140453231611904 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 19:51:56.940704 140453231611904 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1944-hkz7f
I0111 19:53:27.296045 140453231611904 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1944-hkz7f deleted
I0111 19:53:27.296488 140453231611904 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 19:53:27.296767 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-afmvfn87:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1944-hkz7f%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-afmvfn87%22;timeRange=2024-01-11T19:47:11.213124Z%2F2024-01-11T19:51:45.806221Z?project=grpc-testing
I0111 19:53:27.296961 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-5emyk8d4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1944-hkz7f%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-5emyk8d4%22;timeRange=2024-01-11T19:45:45.942281Z%2F2024-01-11T19:53:27.296309Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
I0111 19:53:27.297256 140453231611904 base_testcase.py:52] ----- TestCase __main__.ApiListenerTest.test_api_listener PASSED -----
----------------------------------------------------------------------
Ran 1 test in 537.564s
OK
+ for test in "${test_suites[@]}"
+ run_test change_backend_service_test
+ local test_name=change_backend_service_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test'
+ set -x
+ python -m tests.change_backend_service_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:53:28.657442 139660000452608 xds_k8s_testcase.py:157] ----- Testing ChangeBackendServiceTest -----
I0111 19:53:28.657607 139660000452608 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:53:28.657845 139660000452608 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:53:28.674901 139660000452608 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:53:28.691626 139660000452608 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0111 19:53:28.696330 139660000452608 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1953-20o1e
I0111 19:53:28.991961 139660000452608 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
I0111 19:53:29.436016 139660000452608 xds_k8s_testcase.py:663] Found unused xds port: 49735
I0111 19:53:29.438322 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0111 19:53:29.438541 139660000452608 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1953-20o1e"
I0111 19:53:29.441645 139660000452608 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1953-20o1e
type: GRPC
...
I0111 19:53:30.186767 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002809520-60eb0e6cd72c4-e396e11c-56554943
I0111 19:53:32.504934 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0111 19:53:32.505191 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0111 19:53:32.505293 139660000452608 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1953-20o1e"
I0111 19:53:32.513587 139660000452608 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1953-20o1e
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-1953-20o1e
protocol: GRPC
...
I0111 19:53:33.239455 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002812515-60eb0e6fb2507-0338db9a-500bc107
I0111 19:54:16.438918 139660000452608 traffic_director.py:309] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20240111-1953-20o1e"
I0111 19:54:16.444117 139660000452608 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1953-20o1e
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20240111-1953-20o1e
protocol: GRPC
...
I0111 19:54:17.231704 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002856446-60eb0e9997893-419d1125-1d9cdd97
I0111 19:54:38.813266 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0111 19:54:38.813483 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0111 19:54:38.813588 139660000452608 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1953-20o1e": psm-grpc-server:49735 -> psm-interop-backend-service-20240111-1953-20o1e
I0111 19:54:38.825136 139660000452608 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1953-20o1e
hostRules:
- hosts:
- psm-grpc-server:49735
pathMatcher: psm-interop-path-matcher-20240111-1953-20o1e
name: psm-interop-url-map-20240111-1953-20o1e
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1953-20o1e
name: psm-interop-path-matcher-20240111-1953-20o1e
...
I0111 19:54:39.573349 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002878827-60eb0eaeefa6c-a7d7ca92-dc7ea65d
I0111 19:54:41.939240 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0111 19:54:41.939477 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0111 19:54:41.939605 139660000452608 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1953-20o1e proxy "GRPC" to URL map psm-interop-url-map-20240111-1953-20o1e
I0111 19:54:41.941506 139660000452608 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1953-20o1e
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1953-20o1e
validate_for_proxyless: true
...
I0111 19:54:42.708473 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002881943-60eb0eb1e8929-0bc83180-3af59637
I0111 19:54:44.915428 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0111 19:54:44.915642 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0111 19:54:44.915730 139660000452608 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1953-20o1e" in network "default-vpc": 0.0.0.0:49735 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1953-20o1e
I0111 19:54:44.918292 139660000452608 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1953-20o1e
network: global/networks/default-vpc
portRange: 49735
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1953-20o1e
...
I0111 19:54:45.757917 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002884920-60eb0eb4bf3e7-238ac4a3-f7f3b74a
I0111 19:54:58.990261 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0111 19:54:58.990529 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0111 19:54:58.990638 139660000452608 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1953-20o1e: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:54:58.990908 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22;timeRange=2024-01-11T19:54:58.990740Z%2F2024-01-11T20:24:58.990740Z?project=grpc-testing
I0111 19:54:58.996098 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1953-20o1e
labels:
name: psm-interop-server-20240111-1953-20o1e
owner: xds-k8s-interop-test
...
I0111 19:54:59.145261 139660000452608 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1953-20o1e created
I0111 19:54:59.148380 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1953-20o1e-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:54:59.216807 139660000452608 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:54:59.217024 139660000452608 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1953-20o1e
I0111 19:54:59.311780 139660000452608 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1953-20o1e: neg_name=psm-interop-server-20240111-1953-20o1e-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:54:59.311990 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:55:00.119301 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:55:00.179517 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:55:00.189339 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1953-20o1e-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:55:00.258826 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:55:00.259093 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 19:55:00.366019 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-69cdbd8b49-75d44']
I0111 19:55:00.366358 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-69cdbd8b49-75d44 to start
I0111 19:55:03.606573 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-server-69cdbd8b49-75d44 ready, IP: 10.120.2.171
I0111 19:55:03.606832 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-69cdbd8b49-75d44 to change_backend_service_test/test_app_logs/psm-interop-server-20240111-1953-20o1e_psm-grpc-server-69cdbd8b49-75d44.log
I0111 19:55:03.607274 139659907163712 k8s_log_collector.py:70] Starting log collection thread 139659907163712 for psm-grpc-server-69cdbd8b49-75d44
I0111 19:55:03.607409 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 19:55:08.905764 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 19:55:08.906187 139660000452608 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-1953-20o1e: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:55:08.906420 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22;timeRange=2024-01-11T19:55:08.906299Z%2F2024-01-11T20:25:08.906299Z?project=grpc-testing
I0111 19:55:08.906520 139660000452608 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240111-1953-20o1e
I0111 19:55:08.962622 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:55:09.009193 139660000452608 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 19:55:09.009463 139660000452608 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-1953-20o1e
I0111 19:55:09.104058 139660000452608 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-1953-20o1e: neg_name=psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0111 19:55:09.104266 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:55:09.721852 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:55:09.762687 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 19:55:09.771005 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:55:09.822514 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 19:55:09.822765 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 19:55:09.968732 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:55:09.969035 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-7468b95b99-5rrqx to start
I0111 19:55:12.164124 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-server-alt-7468b95b99-5rrqx ready, IP: 10.120.7.128
I0111 19:55:12.164401 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-7468b95b99-5rrqx to change_backend_service_test/test_app_logs/psm-interop-server-20240111-1953-20o1e_psm-grpc-server-alt-7468b95b99-5rrqx.log
I0111 19:55:12.164910 139659898771008 k8s_log_collector.py:70] Starting log collection thread 139659898771008 for psm-grpc-server-alt-7468b95b99-5rrqx
I0111 19:55:12.165111 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 19:55:19.629064 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 19:55:19.629516 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0111 19:55:19.629632 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0111 19:55:19.666485 139660000452608 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:55:19.789431 139660000452608 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1953-20o1e-psm-grpc-server" in zone us-central1-a
I0111 19:55:19.789646 139660000452608 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1953-20o1e: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1953-20o1e-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server', zone='us-central1-a')}
I0111 19:55:19.794117 139660000452608 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:55:20.596027 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002919796-60eb0ed601e69-45ada39b-b9b58665
I0111 19:56:01.838949 139660000452608 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1953-20o1e to report all backends healthy: ['psm-interop-server-20240111-1953-20o1e-psm-grpc-server']
I0111 19:56:06.356426 139660000452608 compute.py:494] Backend psm-interop-server-20240111-1953-20o1e-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:56:10.399091 139660000452608 traffic_director.py:337] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:56:10.743415 139660000452608 traffic_director.py:340] Loaded NEG "psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt" in zone us-central1-a
I0111 19:56:10.743677 139660000452608 traffic_director.py:347] Adding backends to Backend Service psm-interop-backend-service-alt-20240111-1953-20o1e: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt', zone='us-central1-a')}
I0111 19:56:10.748290 139660000452608 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt
maxRatePerEndpoint: 5
...
I0111 19:56:11.687826 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002970750-60eb0f0699daa-742fae08-0a735aed
I0111 19:56:50.470765 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0111 19:56:50.470988 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0111 19:56:50.471108 139660000452608 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1953-20o1e: server_target=xds:///psm-grpc-server:49735 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 19:56:50.471346 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1953-20o1e%22;timeRange=2024-01-11T19:56:50.471188Z%2F2024-01-11T20:26:50.471188Z?project=grpc-testing
I0111 19:56:50.473242 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1953-20o1e
labels:
name: psm-interop-client-20240111-1953-20o1e
owner: xds-k8s-interop-test
...
I0111 19:56:50.515286 139660000452608 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1953-20o1e created
I0111 19:56:50.515504 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1953-20o1e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:56:51.078962 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:56:51.138321 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:56:51.146969 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1953-20o1e
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:49735"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:56:51.218494 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:56:51.218724 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:56:51.323243 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-bc5fc5bc6-mjjv6']
I0111 19:56:51.323502 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-bc5fc5bc6-mjjv6 to start
I0111 19:56:55.618076 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-client-bc5fc5bc6-mjjv6 ready, IP: 10.120.0.69
I0111 19:56:55.618357 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-bc5fc5bc6-mjjv6 to change_backend_service_test/test_app_logs/psm-interop-client-20240111-1953-20o1e_psm-grpc-client-bc5fc5bc6-mjjv6.log
I0111 19:56:55.618835 139659890378304 k8s_log_collector.py:70] Starting log collection thread 139659890378304 for psm-grpc-client-bc5fc5bc6-mjjv6
I0111 19:56:55.619029 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:56:58.841220 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:56:58.841778 139660000452608 client_app.py:284] [psm-grpc-client-bc5fc5bc6-mjjv6] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:56:58.945901 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:08.991215 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:19.036087 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:29.080876 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:39.125536 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:55.174434 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:58:20.233860 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:58:22.268343 139660000452608 client_app.py:323] [psm-grpc-client-bc5fc5bc6-mjjv6] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:58:22.268630 139660000452608 client_app.py:294] [psm-grpc-client-bc5fc5bc6-mjjv6] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:58:22.268894 139660000452608 client_app.py:244] [psm-grpc-client-bc5fc5bc6-mjjv6] Waiting to report a READY channel to xds:///psm-grpc-server:49735
I0111 19:58:22.300961 139660000452608 client_app.py:371] [psm-grpc-client-bc5fc5bc6-mjjv6] Server channel:
I0111 19:58:22.333167 139660000452608 client_app.py:384] [psm-grpc-client-bc5fc5bc6-mjjv6] Found subchannel in state READY:
I0111 19:58:22.333569 139660000452608 client_app.py:255] [psm-grpc-client-bc5fc5bc6-mjjv6] Channel to xds:///psm-grpc-server:49735 transitioned to state READY:
I0111 19:58:22.333703 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0111 19:58:22.333796 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0111 19:58:22.333996 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:58:22.368292 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0111 19:58:22.368509 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0111 19:58:22.368712 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:26.382597 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:26.382882 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0111 19:58:26.382986 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0111 19:58:26.383072 139660000452608 traffic_director.py:499] Patching URL map "psm-interop-url-map-20240111-1953-20o1e": psm-grpc-server:49735 -> psm-interop-backend-service-alt-20240111-1953-20o1e
I0111 19:58:26.388541 139660000452608 compute.py:544] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240111-1953-20o1e
hostRules:
- hosts:
- psm-grpc-server:49735
pathMatcher: psm-interop-path-matcher-20240111-1953-20o1e
name: psm-interop-url-map-20240111-1953-20o1e
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240111-1953-20o1e
name: psm-interop-path-matcher-20240111-1953-20o1e
...
I0111 19:58:27.381674 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003106391-60eb0f87f5552-fadee9af-ea78c0f9
I0111 19:58:38.370954 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:38.371234 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:42.383072 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:42.383684 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7468b95b99-5rrqx' not found in {'psm-grpc-server-69cdbd8b49-75d44': 100} : Server psm-grpc-server-alt-7468b95b99-5rrqx did not receive RPCs.
I0111 19:58:43.385011 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:43.385346 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:47.382387 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:47.382673 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7468b95b99-5rrqx' not found in {'psm-grpc-server-69cdbd8b49-75d44': 100} : Server psm-grpc-server-alt-7468b95b99-5rrqx did not receive RPCs.
I0111 19:58:48.383917 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:48.384191 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:53.430449 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7468b95b99-5rrqx: 92
psm-grpc-server-69cdbd8b49-75d44: 8
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7468b95b99-5rrqx: 92
psm-grpc-server-69cdbd8b49-75d44: 8
I0111 19:58:53.430794 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cdbd8b49-75d44' not found in ['psm-grpc-server-alt-7468b95b99-5rrqx'] : Unexpected server psm-grpc-server-69cdbd8b49-75d44 received RPCs.
I0111 19:58:54.432078 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:54.432406 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:58.462263 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7468b95b99-5rrqx: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7468b95b99-5rrqx: 100
I0111 19:58:58.462595 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0111 19:58:58.462719 139660000452608 xds_k8s_testcase.py:687] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0111 19:58:58.685106 139660000452608 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1953-20o1e"
I0111 19:58:59.281219 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003138689-60eb0fa6c29bd-6ff68c6f-72f373a2
I0111 19:59:12.473219 139660000452608 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1953-20o1e"
I0111 19:59:12.586460 139660000452608 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1953-20o1e"
I0111 19:59:12.995800 139660000452608 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1953-20o1e"
I0111 19:59:13.370333 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003152999-60eb0fb468417-5dafcb4d-334ae9a4
I0111 19:59:15.640515 139660000452608 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1953-20o1e"
I0111 19:59:16.058938 139660000452608 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1953-20o1e"
I0111 19:59:16.717587 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003156064-60eb0fb75491b-5a002d0c-fac71bea
I0111 19:59:18.982602 139660000452608 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1953-20o1e"
I0111 19:59:19.407786 139660000452608 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1953-20o1e"
I0111 19:59:19.989174 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003159413-60eb0fba863e8-6c23bf6e-7a192969
I0111 19:59:45.983453 139660000452608 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1953-20o1e"
I0111 19:59:46.640601 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003185988-60eb0fd3de47d-5dc40e73-8dfbbed0
I0111 20:00:16.912211 139660000452608 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1953-20o1e"
I0111 20:00:17.413522 139660000452608 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1953-20o1e"
I0111 20:00:17.975465 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003217570-60eb0ff1fc919-2ce2d0b9-1d934ce6
I0111 20:00:22.434183 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:00:32.613351 139660000452608 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:00:32.613661 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1953-20o1e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:00:33.190623 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:00:33.303453 139660000452608 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:00:33.303676 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1953-20o1e
I0111 20:01:03.524804 139660000452608 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1953-20o1e deleted
I0111 20:01:03.525126 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:01:13.698527 139660000452608 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:01:13.698828 139660000452608 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:01:13.815351 139660000452608 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:01:13.815662 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:01:14.644493 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:01:14.755231 139660000452608 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:01:14.755490 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1953-20o1e
I0111 20:02:15.002261 139660000452608 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1953-20o1e deleted
I0111 20:02:15.002571 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:02:15.058662 139660000452608 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:02:15.115298 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
W0111 20:02:17.683681 139660000452608 http.py:163] Sleeping 0.85 seconds before retry 1 of 5 for request: POST https://iam.googleapis.com/v1/projects/grpc-testing/serviceAccounts/xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com:setIamPolicy?alt=json, after 502
I0111 20:02:18.956943 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:02:19.043891 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1953-20o1e
I0111 20:02:19.079876 139660000452608 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:02:19.080252 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-6gyraz8u:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1953-20o1e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-6gyraz8u%22;timeRange=2024-01-11T19:56:50.471180Z%2F2024-01-11T20:01:03.525094Z?project=grpc-testing
I0111 20:02:19.080415 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-snef9hve:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-snef9hve%22;timeRange=2024-01-11T19:54:58.990732Z%2F2024-01-11T20:02:15.002541Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
I0111 20:02:19.080672 139660000452608 base_testcase.py:52] ----- TestCase __main__.ChangeBackendServiceTest.test_change_backend_service PASSED -----
----------------------------------------------------------------------
Ran 1 test in 530.426s
OK
+ for test in "${test_suites[@]}"
+ run_test custom_lb_test
+ local test_name=custom_lb_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test'
+ set -x
+ python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:02:20.479438 139846197112832 xds_k8s_testcase.py:157] ----- Testing CustomLbTest -----
I0111 20:02:20.479620 139846197112832 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:02:20.479863 139846197112832 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:02:20.496662 139846197112832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:02:20.513510 139846197112832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0111 20:02:20.518397 139846197112832 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2002-cg3w0
I0111 20:02:20.852132 139846197112832 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
I0111 20:02:21.881125 139846197112832 xds_k8s_testcase.py:663] Found unused xds port: 33446
I0111 20:02:21.881491 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0111 20:02:21.881592 139846197112832 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2002-cg3w0"
I0111 20:02:21.884764 139846197112832 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2002-cg3w0
type: GRPC
...
I0111 20:02:23.104360 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003341932-60eb10689669d-3d97c978-4191196f
I0111 20:02:25.461221 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0111 20:02:25.461428 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0111 20:02:25.461530 139846197112832 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2002-cg3w0"
I0111 20:02:25.470824 139846197112832 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2002-cg3w0
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240111-2002-cg3w0
protocol: GRPC
...
I0111 20:02:26.325885 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003345474-60eb106bf73a5-04d219e9-7cd47553
I0111 20:02:48.126820 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0111 20:02:48.127070 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0111 20:02:48.127166 139846197112832 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2002-cg3w0": psm-grpc-server:33446 -> psm-interop-backend-service-20240111-2002-cg3w0
I0111 20:02:48.138430 139846197112832 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2002-cg3w0
hostRules:
- hosts:
- psm-grpc-server:33446
pathMatcher: psm-interop-path-matcher-20240111-2002-cg3w0
name: psm-interop-url-map-20240111-2002-cg3w0
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2002-cg3w0
name: psm-interop-path-matcher-20240111-2002-cg3w0
...
I0111 20:02:48.829525 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003368140-60eb108194f01-3d2a7cee-9d5285b6
I0111 20:02:51.216289 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0111 20:02:51.216522 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0111 20:02:51.216617 139846197112832 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2002-cg3w0 proxy "GRPC" to URL map psm-interop-url-map-20240111-2002-cg3w0
I0111 20:02:51.218715 139846197112832 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2002-cg3w0
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2002-cg3w0
validate_for_proxyless: true
...
I0111 20:02:51.782716 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003371220-60eb108484e67-9b1a7184-556228ca
I0111 20:02:54.049887 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0111 20:02:54.050159 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0111 20:02:54.050268 139846197112832 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2002-cg3w0" in network "default-vpc": 0.0.0.0:33446 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2002-cg3w0
I0111 20:02:54.052963 139846197112832 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2002-cg3w0
network: global/networks/default-vpc
portRange: 33446
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2002-cg3w0
...
I0111 20:02:55.107727 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003374055-60eb108738ea8-017e446e-62283dcc
I0111 20:03:08.007846 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0111 20:03:08.008091 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0111 20:03:08.008215 139846197112832 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2002-cg3w0: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:03:08.008456 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2002-cg3w0%22;timeRange=2024-01-11T20:03:08.008295Z%2F2024-01-11T20:33:08.008295Z?project=grpc-testing
I0111 20:03:08.013872 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2002-cg3w0
labels:
name: psm-interop-server-20240111-2002-cg3w0
owner: xds-k8s-interop-test
...
I0111 20:03:08.175436 139846197112832 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2002-cg3w0 created
I0111 20:03:08.178532 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2002-cg3w0-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:03:08.252398 139846197112832 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:03:08.252864 139846197112832 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2002-cg3w0
I0111 20:03:08.346837 139846197112832 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2002-cg3w0: neg_name=psm-interop-server-20240111-2002-cg3w0-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:03:08.347062 139846197112832 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2002-cg3w0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:03:09.261242 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:03:09.324512 139846197112832 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:03:09.334034 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2002-cg3w0-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:03:09.411838 139846197112832 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:03:09.412075 139846197112832 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:03:09.522282 139846197112832 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-758558bd4-5mr5j']
I0111 20:03:09.522584 139846197112832 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-758558bd4-5mr5j to start
I0111 20:03:12.781654 139846197112832 k8s_base_runner.py:820] Pod psm-grpc-server-758558bd4-5mr5j ready, IP: 10.120.2.173
I0111 20:03:12.781950 139846197112832 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-758558bd4-5mr5j to custom_lb_test/test_app_logs/psm-interop-server-20240111-2002-cg3w0_psm-grpc-server-758558bd4-5mr5j.log
I0111 20:03:12.782462 139846103852608 k8s_log_collector.py:70] Starting log collection thread 139846103852608 for psm-grpc-server-758558bd4-5mr5j
I0111 20:03:12.782639 139846197112832 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:03:19.195806 139846197112832 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:03:19.196253 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0111 20:03:19.196375 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0111 20:03:19.233335 139846197112832 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:03:19.357439 139846197112832 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2002-cg3w0-psm-grpc-server" in zone us-central1-a
I0111 20:03:19.357697 139846197112832 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2002-cg3w0: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2002-cg3w0-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2002-cg3w0-psm-grpc-server', zone='us-central1-a')}
I0111 20:03:19.362277 139846197112832 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2002-cg3w0-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:03:20.168581 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003399364-60eb109f5bf41-0389b081-d04ab39b
I0111 20:04:05.214567 139846197112832 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2002-cg3w0 to report all backends healthy: ['psm-interop-server-20240111-2002-cg3w0-psm-grpc-server']
I0111 20:04:05.405455 139846197112832 compute.py:494] Backend psm-interop-server-20240111-2002-cg3w0-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:04:09.410336 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0111 20:04:09.410570 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0111 20:04:09.410681 139846197112832 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2002-cg3w0: server_target=xds:///psm-grpc-server:33446 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:04:09.410934 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2002-cg3w0%22;timeRange=2024-01-11T20:04:09.410787Z%2F2024-01-11T20:34:09.410787Z?project=grpc-testing
I0111 20:04:09.413172 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2002-cg3w0
labels:
name: psm-interop-client-20240111-2002-cg3w0
owner: xds-k8s-interop-test
...
I0111 20:04:09.455238 139846197112832 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2002-cg3w0 created
I0111 20:04:09.455470 139846197112832 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2002-cg3w0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:04:10.129106 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:04:10.189777 139846197112832 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:04:10.198081 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2002-cg3w0
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:33446"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:04:10.272607 139846197112832 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:04:10.272888 139846197112832 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:04:10.442100 139846197112832 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-f45fdd996-wnsn7']
I0111 20:04:10.442408 139846197112832 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-f45fdd996-wnsn7 to start
I0111 20:04:14.738801 139846197112832 k8s_base_runner.py:820] Pod psm-grpc-client-f45fdd996-wnsn7 ready, IP: 10.120.7.129
I0111 20:04:14.739075 139846197112832 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-f45fdd996-wnsn7 to custom_lb_test/test_app_logs/psm-interop-client-20240111-2002-cg3w0_psm-grpc-client-f45fdd996-wnsn7.log
I0111 20:04:14.739586 139846095459904 k8s_log_collector.py:70] Starting log collection thread 139846095459904 for psm-grpc-client-f45fdd996-wnsn7
I0111 20:04:14.739763 139846197112832 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:04:17.964606 139846197112832 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:04:17.965142 139846197112832 client_app.py:284] [psm-grpc-client-f45fdd996-wnsn7] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:04:18.069370 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:28.115473 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:38.160590 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:48.206417 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:58.251483 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:14.301072 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:39.361058 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:41.396371 139846197112832 client_app.py:323] [psm-grpc-client-f45fdd996-wnsn7] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:05:41.396664 139846197112832 client_app.py:294] [psm-grpc-client-f45fdd996-wnsn7] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:05:41.396916 139846197112832 client_app.py:244] [psm-grpc-client-f45fdd996-wnsn7] Waiting to report a READY channel to xds:///psm-grpc-server:33446
I0111 20:05:41.429580 139846197112832 client_app.py:371] [psm-grpc-client-f45fdd996-wnsn7] Server channel:
I0111 20:05:41.462421 139846197112832 client_app.py:384] [psm-grpc-client-f45fdd996-wnsn7] Found subchannel in state READY:
I0111 20:05:41.462860 139846197112832 client_app.py:255] [psm-grpc-client-f45fdd996-wnsn7] Channel to xds:///psm-grpc-server:33446 transitioned to state READY:
I0111 20:05:41.463012 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0111 20:05:41.463092 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0111 20:05:41.463308 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 20:05:41.498455 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0111 20:05:41.498718 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0111 20:05:41.498947 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0111 20:05:51.543184 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0111 20:05:51.579789 139846197112832 xds_k8s_testcase.py:398] [psm-grpc-client-f45fdd996-wnsn7] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 251
result:
(15, DATA_LOSS): 252
I0111 20:05:51.580506 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0111 20:05:51.580660 139846197112832 xds_k8s_testcase.py:687] ----- TestMethod __main__.CustomLbTest.test_custom_lb_config teardown -----
I0111 20:05:51.806437 139846197112832 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2002-cg3w0"
I0111 20:05:52.552935 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003551810-60eb1130be3e5-af599ec5-4fc0a684
I0111 20:06:16.626568 139846197112832 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2002-cg3w0"
I0111 20:06:16.743684 139846197112832 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2002-cg3w0"
I0111 20:06:17.198181 139846197112832 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2002-cg3w0"
I0111 20:06:17.844416 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003577201-60eb1148f5440-9e68ded1-3d67c5ba
I0111 20:06:20.105163 139846197112832 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2002-cg3w0"
I0111 20:06:20.364686 139846197112832 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2002-cg3w0"
I0111 20:06:21.144745 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003580370-60eb114bfaf7c-f3900da1-dff61e6b
I0111 20:06:25.723691 139846197112832 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2002-cg3w0"
I0111 20:06:26.205460 139846197112832 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2002-cg3w0"
I0111 20:06:26.848117 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003586210-60eb11518cc78-fbd9abe1-3803ec1d
I0111 20:06:52.742841 139846197112832 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2002-cg3w0"
I0111 20:06:53.194561 139846197112832 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2002-cg3w0"
I0111 20:06:53.620371 139846197112832 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2002-cg3w0"
I0111 20:06:54.199699 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003613692-60eb116bc2371-69d84b05-95d6b990
I0111 20:06:58.568403 139846197112832 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:07:08.751038 139846197112832 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:07:08.751373 139846197112832 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2002-cg3w0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:07:09.397170 139846197112832 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:07:09.509819 139846197112832 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:07:09.510140 139846197112832 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2002-cg3w0
I0111 20:07:39.693346 139846197112832 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2002-cg3w0 deleted
I0111 20:07:39.693661 139846197112832 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:07:49.930631 139846197112832 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:07:49.930931 139846197112832 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:07:50.054333 139846197112832 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:07:50.054588 139846197112832 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2002-cg3w0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:07:50.599160 139846197112832 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:07:50.726630 139846197112832 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:07:50.726869 139846197112832 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2002-cg3w0
I0111 20:08:20.941802 139846197112832 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2002-cg3w0 deleted
I0111 20:08:20.942281 139846197112832 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:08:20.942540 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-zkqpsbgt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2002-cg3w0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-zkqpsbgt%22;timeRange=2024-01-11T20:04:09.410780Z%2F2024-01-11T20:07:39.693623Z?project=grpc-testing
I0111 20:08:20.942691 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-kn89amww:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2002-cg3w0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-kn89amww%22;timeRange=2024-01-11T20:03:08.008288Z%2F2024-01-11T20:08:20.942112Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0111 20:08:20.942970 139846197112832 base_testcase.py:52] ----- TestCase __main__.CustomLbTest.test_custom_lb_config PASSED -----
----------------------------------------------------------------------
Ran 1 test in 360.466s
OK
+ for test in "${test_suites[@]}"
+ run_test failover_test
+ local test_name=failover_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test'
+ set -x
+ python -m tests.failover_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:08:22.387956 139913641447424 xds_k8s_testcase.py:157] ----- Testing FailoverTest -----
I0111 20:08:22.388122 139913641447424 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:08:22.388356 139913641447424 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:08:22.406023 139913641447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:08:22.423303 139913641447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0111 20:08:22.428613 139913641447424 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2008-boazh
I0111 20:08:22.736891 139913641447424 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
I0111 20:08:23.120027 139913641447424 xds_k8s_testcase.py:663] Found unused xds port: 6919
I0111 20:08:23.122310 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0111 20:08:23.122454 139913641447424 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2008-boazh"
I0111 20:08:23.125547 139913641447424 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2008-boazh
type: GRPC
...
I0111 20:08:23.759954 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003703169-60eb11c11731b-416258f6-6865d981
I0111 20:08:25.982944 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0111 20:08:25.983177 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0111 20:08:25.983278 139913641447424 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2008-boazh"
I0111 20:08:25.991434 139913641447424 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2008-boazh
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2008-boazh
protocol: GRPC
...
I0111 20:08:26.876192 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003705994-60eb11c3c8e4d-ba3342e5-f5545b95
I0111 20:09:05.874300 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0111 20:09:05.874537 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0111 20:09:05.874629 139913641447424 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2008-boazh": psm-grpc-server:6919 -> psm-interop-backend-service-20240111-2008-boazh
I0111 20:09:05.886891 139913641447424 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2008-boazh
hostRules:
- hosts:
- psm-grpc-server:6919
pathMatcher: psm-interop-path-matcher-20240111-2008-boazh
name: psm-interop-url-map-20240111-2008-boazh
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2008-boazh
name: psm-interop-path-matcher-20240111-2008-boazh
...
I0111 20:09:06.542138 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003745890-60eb11e9d5157-0fd663ea-4f83cadb
I0111 20:09:08.801229 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0111 20:09:08.801444 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0111 20:09:08.801552 139913641447424 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2008-boazh proxy "GRPC" to URL map psm-interop-url-map-20240111-2008-boazh
I0111 20:09:08.803504 139913641447424 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2008-boazh
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2008-boazh
validate_for_proxyless: true
...
I0111 20:09:09.283087 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003748805-60eb11ec9cc09-812eb62f-879a1c51
I0111 20:09:11.615242 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0111 20:09:11.615501 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0111 20:09:11.615630 139913641447424 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2008-boazh" in network "default-vpc": 0.0.0.0:6919 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2008-boazh
I0111 20:09:11.618214 139913641447424 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2008-boazh
network: global/networks/default-vpc
portRange: 6919
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2008-boazh
...
I0111 20:09:12.458112 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003751620-60eb11ef4bfc4-2168049e-fd4a833f
I0111 20:09:25.391936 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0111 20:09:25.392173 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0111 20:09:25.392290 139913641447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2008-boazh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0111 20:09:25.392578 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22;timeRange=2024-01-11T20:09:25.392380Z%2F2024-01-11T20:39:25.392380Z?project=grpc-testing
I0111 20:09:25.397961 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2008-boazh
labels:
name: psm-interop-server-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:09:25.536939 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2008-boazh created
I0111 20:09:25.539982 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2008-boazh-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:09:25.614156 139913641447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:09:25.614384 139913641447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2008-boazh
I0111 20:09:25.706863 139913641447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2008-boazh: neg_name=psm-interop-server-20240111-2008-boazh-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:09:25.707097 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:09:26.505964 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:09:26.576163 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:09:26.585172 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2008-boazh-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:09:26.660861 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:09:26.661137 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0111 20:09:26.815884 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:09:26.816328 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-6l52f to start
I0111 20:09:29.005968 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-6l52f ready, IP: 10.120.0.71
I0111 20:09:29.006224 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-6l52f to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-6l52f.log
I0111 20:09:29.006721 139913548260928 k8s_log_collector.py:70] Starting log collection thread 139913548260928 for psm-grpc-server-76f54665-6l52f
I0111 20:09:29.006996 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-hjgdh to start
I0111 20:09:30.225463 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-hjgdh ready, IP: 10.120.3.103
I0111 20:09:30.225731 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-hjgdh to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-hjgdh.log
I0111 20:09:30.226195 139913539868224 k8s_log_collector.py:70] Starting log collection thread 139913539868224 for psm-grpc-server-76f54665-hjgdh
I0111 20:09:30.226339 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-zsx45 to start
I0111 20:09:30.400224 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-zsx45 ready, IP: 10.120.2.174
I0111 20:09:30.400607 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-zsx45 to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-zsx45.log
I0111 20:09:30.401073 139913458611776 k8s_log_collector.py:70] Starting log collection thread 139913458611776 for psm-grpc-server-76f54665-zsx45
I0111 20:09:30.401249 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0111 20:09:36.738187 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0111 20:09:36.738750 139913641447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-2008-boazh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:09:36.739004 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22;timeRange=2024-01-11T20:09:36.738882Z%2F2024-01-11T20:39:36.738882Z?project=grpc-testing
I0111 20:09:36.740842 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2008-boazh
labels:
name: psm-interop-server-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:09:36.990764 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2008-boazh created
I0111 20:09:36.993801 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:09:37.121321 139913641447424 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 20:09:37.121567 139913641447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-2008-boazh
I0111 20:09:37.280402 139913641447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-2008-boazh: neg_name=psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt, port=8080, zones=['us-west1-b']
I0111 20:09:37.280618 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:09:37.931873 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:09:38.023608 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 20:09:38.031785 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:09:38.137392 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 20:09:38.137653 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 20:09:38.316219 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:09:38.316516 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-6c6cc65f86-v4lv8 to start
I0111 20:09:44.951710 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-alt-6c6cc65f86-v4lv8 ready, IP: 10.108.6.100
I0111 20:09:44.951983 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-6c6cc65f86-v4lv8 to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-alt-6c6cc65f86-v4lv8.log
I0111 20:09:44.952458 139913450219072 k8s_log_collector.py:70] Starting log collection thread 139913450219072 for psm-grpc-server-alt-6c6cc65f86-v4lv8
I0111 20:09:44.952631 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 20:09:49.554591 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 20:09:49.555037 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0111 20:09:49.555170 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0111 20:09:49.593019 139913641447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:09:49.890694 139913641447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2008-boazh-psm-grpc-server" in zone us-central1-a
I0111 20:09:49.890915 139913641447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2008-boazh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server', zone='us-central1-a')}
I0111 20:09:49.895409 139913641447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server
maxRatePerEndpoint: 100
...
I0111 20:09:51.177198 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003789898-60eb1213cd43e-d9c184c7-5b75c0b3
I0111 20:10:40.993617 139913641447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2008-boazh to report all backends healthy: ['psm-interop-server-20240111-2008-boazh-psm-grpc-server']
I0111 20:10:41.182751 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:10:45.259872 139913641447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:10:45.734279 139913641447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt" in zone us-west1-b
I0111 20:10:45.734519 139913641447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2008-boazh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server', zone='us-central1-a')}
I0111 20:10:45.739555 139913641447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server
maxRatePerEndpoint: 100
...
I0111 20:10:47.732389 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003845741-60eb12490ed43-b8015dee-7a3ac74b
I0111 20:11:39.755125 139913641447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2008-boazh to report all backends healthy: ['psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', 'psm-interop-server-20240111-2008-boazh-psm-grpc-server']
I0111 20:11:40.637006 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt in zone us-west1-b reported healthy
I0111 20:11:45.216178 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:11:49.221073 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0111 20:11:49.221302 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0111 20:11:49.221412 139913641447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2008-boazh: server_target=xds:///psm-grpc-server:6919 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:11:49.221659 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2008-boazh%22;timeRange=2024-01-11T20:11:49.221493Z%2F2024-01-11T20:41:49.221493Z?project=grpc-testing
I0111 20:11:49.223608 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2008-boazh
labels:
name: psm-interop-client-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:11:49.368944 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2008-boazh created
I0111 20:11:49.369181 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2008-boazh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:11:49.956958 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:11:50.015135 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:11:50.023332 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2008-boazh
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:6919"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:11:50.107287 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:11:50.107581 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:11:50.210240 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6b767fbc77-vsz7v']
I0111 20:11:50.210525 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6b767fbc77-vsz7v to start
I0111 20:11:53.439570 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-client-6b767fbc77-vsz7v ready, IP: 10.120.8.64
I0111 20:11:53.439865 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6b767fbc77-vsz7v to failover_test/test_app_logs/psm-interop-client-20240111-2008-boazh_psm-grpc-client-6b767fbc77-vsz7v.log
I0111 20:11:53.440364 139913441826368 k8s_log_collector.py:70] Starting log collection thread 139913441826368 for psm-grpc-client-6b767fbc77-vsz7v
I0111 20:11:53.440528 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:11:54.576689 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:11:54.577218 139913641447424 client_app.py:284] [psm-grpc-client-6b767fbc77-vsz7v] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:11:54.680208 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:04.724061 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:14.768380 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:24.812790 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:34.857550 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:50.906437 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:13:15.966430 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:13:18.001230 139913641447424 client_app.py:323] [psm-grpc-client-6b767fbc77-vsz7v] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:13:18.001528 139913641447424 client_app.py:294] [psm-grpc-client-6b767fbc77-vsz7v] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:13:18.001826 139913641447424 client_app.py:244] [psm-grpc-client-6b767fbc77-vsz7v] Waiting to report a READY channel to xds:///psm-grpc-server:6919
I0111 20:13:18.034170 139913641447424 client_app.py:371] [psm-grpc-client-6b767fbc77-vsz7v] Server channel:
I0111 20:13:18.066734 139913641447424 client_app.py:384] [psm-grpc-client-6b767fbc77-vsz7v] Found subchannel in state READY:
I0111 20:13:18.067214 139913641447424 client_app.py:255] [psm-grpc-client-6b767fbc77-vsz7v] Channel to xds:///psm-grpc-server:6919 transitioned to state READY:
I0111 20:13:18.067361 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0111 20:13:18.067438 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0111 20:13:18.067713 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:18.102243 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0111 20:13:18.102484 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0111 20:13:18.102802 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:18.103056 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:22.121807 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 34
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 34
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 33
I0111 20:13:22.122264 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0111 20:13:22.122374 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0111 20:13:22.122452 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-6l52f] >> Setting health status to NOT_SERVING
I0111 20:13:22.122793 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:13:22.219638 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:22.251611 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-6l52f] << Health status status: NOT_SERVING
I0111 20:13:22.252102 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:22.252297 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:26.281747 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 34
I0111 20:13:26.282323 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:27.283626 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:27.283970 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:31.282632 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 34
psm-grpc-server-76f54665-hjgdh: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 34
psm-grpc-server-76f54665-hjgdh: 33
I0111 20:13:31.283039 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:32.284381 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:32.284735 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:36.281849 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 4
psm-grpc-server-76f54665-zsx45: 48
psm-grpc-server-76f54665-hjgdh: 48
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 4
psm-grpc-server-76f54665-zsx45: 48
psm-grpc-server-76f54665-hjgdh: 48
I0111 20:13:36.282249 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:37.283644 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:37.284029 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:41.281406 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:41.281740 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0111 20:13:41.281833 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0111 20:13:41.281903 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-hjgdh] >> Setting health status to NOT_SERVING
I0111 20:13:41.282249 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:13:41.378674 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:41.410757 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-hjgdh] << Health status status: NOT_SERVING
I0111 20:13:41.411232 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:41.411411 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:45.442096 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:45.442679 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-6c6cc65f86-v4lv8' not found in {'psm-grpc-server-76f54665-zsx45': 50, 'psm-grpc-server-76f54665-hjgdh': 50} : Server psm-grpc-server-alt-6c6cc65f86-v4lv8 did not receive RPCs.
I0111 20:13:46.444025 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:46.444355 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:50.441465 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:50.441771 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-6c6cc65f86-v4lv8' not found in {'psm-grpc-server-76f54665-zsx45': 50, 'psm-grpc-server-76f54665-hjgdh': 50} : Server psm-grpc-server-alt-6c6cc65f86-v4lv8 did not receive RPCs.
I0111 20:13:51.443064 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:51.443352 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:55.441804 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 60
psm-grpc-server-76f54665-hjgdh: 16
psm-grpc-server-alt-6c6cc65f86-v4lv8: 24
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 60
psm-grpc-server-76f54665-hjgdh: 16
psm-grpc-server-alt-6c6cc65f86-v4lv8: 24
I0111 20:13:55.442188 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-hjgdh' not found in ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-hjgdh received RPCs.
I0111 20:13:56.443488 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:56.443871 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:00.442048 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
I0111 20:14:00.442391 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0111 20:14:00.442486 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0111 20:14:00.442556 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-zsx45] >> Setting health status to NOT_SERVING
I0111 20:14:00.442892 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:00.539745 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:00.572016 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-zsx45] << Health status status: NOT_SERVING
I0111 20:14:00.572509 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:00.572736 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:04.633217 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
I0111 20:14:04.633792 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-zsx45' not found in ['psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-zsx45 received RPCs.
I0111 20:14:05.635100 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:05.635426 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:09.673938 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 15
psm-grpc-server-alt-6c6cc65f86-v4lv8: 85
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 15
psm-grpc-server-alt-6c6cc65f86-v4lv8: 85
I0111 20:14:09.674326 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-zsx45' not found in ['psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-zsx45 received RPCs.
I0111 20:14:10.675648 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:10.676008 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:14.712773 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
I0111 20:14:14.713163 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0111 20:14:14.713267 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0111 20:14:14.713340 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-6l52f] >> Setting health status to SERVING
I0111 20:14:14.713447 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.747150 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.779174 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-6l52f] << Health status status: SERVING
I0111 20:14:14.779399 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-hjgdh] >> Setting health status to SERVING
I0111 20:14:14.779556 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.811865 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.843859 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-hjgdh] << Health status status: SERVING
I0111 20:14:14.844110 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-zsx45] >> Setting health status to SERVING
I0111 20:14:14.844231 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.878186 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.910275 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-zsx45] << Health status status: SERVING
I0111 20:14:14.910741 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:14.910954 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:18.953256 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
I0111 20:14:18.953824 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in {'psm-grpc-server-alt-6c6cc65f86-v4lv8': 100} : Server psm-grpc-server-76f54665-6l52f did not receive RPCs.
I0111 20:14:19.955106 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:19.955458 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:23.961477 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 21
psm-grpc-server-alt-6c6cc65f86-v4lv8: 79
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 21
psm-grpc-server-alt-6c6cc65f86-v4lv8: 79
I0111 20:14:23.961865 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-hjgdh' not found in {'psm-grpc-server-76f54665-6l52f': 21, 'psm-grpc-server-alt-6c6cc65f86-v4lv8': 79} : Server psm-grpc-server-76f54665-hjgdh did not receive RPCs.
I0111 20:14:24.963189 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:24.963607 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:28.962077 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 41
psm-grpc-server-76f54665-zsx45: 40
psm-grpc-server-76f54665-hjgdh: 19
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 41
psm-grpc-server-76f54665-zsx45: 40
psm-grpc-server-76f54665-hjgdh: 19
I0111 20:14:28.962437 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0111 20:14:28.962561 139913641447424 xds_k8s_testcase.py:687] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0111 20:14:29.223731 139913641447424 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2008-boazh"
I0111 20:14:29.937295 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004069228-60eb131e311bd-f73dc2dc-74af7019
I0111 20:14:43.033998 139913641447424 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2008-boazh"
I0111 20:14:43.141362 139913641447424 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2008-boazh"
I0111 20:14:43.597512 139913641447424 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2008-boazh"
I0111 20:14:44.139784 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004083600-60eb132be5d8d-84bca90d-4d56dbce
I0111 20:14:46.400274 139913641447424 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2008-boazh"
I0111 20:14:46.738552 139913641447424 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2008-boazh"
I0111 20:14:47.456656 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004086743-60eb132ee5327-d77472f9-c7e90255
I0111 20:14:49.844459 139913641447424 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2008-boazh"
I0111 20:14:50.294652 139913641447424 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2008-boazh"
I0111 20:14:50.891716 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004090300-60eb13324985b-2483acc5-f00b137c
I0111 20:15:27.600957 139913641447424 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2008-boazh"
I0111 20:15:27.942696 139913641447424 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2008-boazh"
I0111 20:15:28.400301 139913641447424 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2008-boazh"
I0111 20:15:28.865722 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004128476-60eb1356b1ba5-9b3e8adf-fd8bdb5c
I0111 20:15:33.329291 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:15:43.526966 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:15:43.527270 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2008-boazh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:15:44.061680 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:15:44.178549 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:15:44.178780 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2008-boazh
I0111 20:16:14.361317 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2008-boazh deleted
I0111 20:16:14.361594 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:16:24.535983 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:16:24.536315 139913641447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:16:24.650049 139913641447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:16:24.650289 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:16:25.266630 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:16:25.389813 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:16:25.390047 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2008-boazh
I0111 20:18:55.918070 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2008-boazh deleted
I0111 20:18:55.918290 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:19:06.362561 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-server-alt deleted
I0111 20:19:06.362869 139913641447424 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:19:06.550542 139913641447424 k8s_base_runner.py:739] Service psm-grpc-server-alt deleted
I0111 20:19:06.550800 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:19:07.112985 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:19:07.296326 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-server-alt deleted
I0111 20:19:07.296561 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2008-boazh
I0111 20:21:07.905254 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2008-boazh deleted
I0111 20:21:07.905699 139913641447424 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:21:07.905927 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-gx2jhwbf:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2008-boazh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-gx2jhwbf%22;timeRange=2024-01-11T20:11:49.221485Z%2F2024-01-11T20:16:14.361563Z?project=grpc-testing
I0111 20:21:07.906091 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-4rktopzs:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-4rktopzs%22;timeRange=2024-01-11T20:09:25.392372Z%2F2024-01-11T20:18:55.918259Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
I0111 20:21:07.906363 139913641447424 base_testcase.py:52] ----- TestCase __main__.FailoverTest.test_failover PASSED -----
----------------------------------------------------------------------
Ran 1 test in 765.521s
OK
+ for test in "${test_suites[@]}"
+ run_test outlier_detection_test
+ local test_name=outlier_detection_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test'
+ set -x
+ python -m tests.outlier_detection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:21:09.368448 140660168441856 xds_k8s_testcase.py:157] ----- Testing OutlierDetectionTest -----
I0111 20:21:09.368624 140660168441856 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:21:09.368925 140660168441856 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:21:09.386436 140660168441856 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:21:09.403961 140660168441856 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0111 20:21:09.408847 140660168441856 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2021-7a5zc
I0111 20:21:09.713453 140660168441856 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
I0111 20:21:10.025703 140660168441856 xds_k8s_testcase.py:663] Found unused xds port: 58833
I0111 20:21:10.026110 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0111 20:21:10.026195 140660168441856 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2021-7a5zc"
I0111 20:21:10.029386 140660168441856 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2021-7a5zc
type: GRPC
...
I0111 20:21:10.795434 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004470091-60eb149c7be36-ab19e17e-ba69098e
I0111 20:21:13.144676 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0111 20:21:13.144854 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0111 20:21:13.144926 140660168441856 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2021-7a5zc"
I0111 20:21:13.153614 140660168441856 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2021-7a5zc
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2021-7a5zc
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...
I0111 20:21:13.803114 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004473155-60eb149f68097-ee1e17fd-8daefb97
I0111 20:22:03.405103 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0111 20:22:03.405283 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0111 20:22:03.405356 140660168441856 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2021-7a5zc": psm-grpc-server:58833 -> psm-interop-backend-service-20240111-2021-7a5zc
I0111 20:22:03.416995 140660168441856 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2021-7a5zc
hostRules:
- hosts:
- psm-grpc-server:58833
pathMatcher: psm-interop-path-matcher-20240111-2021-7a5zc
name: psm-interop-url-map-20240111-2021-7a5zc
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2021-7a5zc
name: psm-interop-path-matcher-20240111-2021-7a5zc
...
I0111 20:22:04.110036 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004523418-60eb14cf574da-e90db041-8b4dc542
I0111 20:22:06.475903 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0111 20:22:06.476128 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0111 20:22:06.476220 140660168441856 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2021-7a5zc proxy "GRPC" to URL map psm-interop-url-map-20240111-2021-7a5zc
I0111 20:22:06.478308 140660168441856 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2021-7a5zc
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2021-7a5zc
validate_for_proxyless: true
...
I0111 20:22:07.033910 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004526480-60eb14d242c44-d71443ed-dc74897c
I0111 20:22:09.435275 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0111 20:22:09.435516 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0111 20:22:09.435667 140660168441856 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2021-7a5zc" in network "default-vpc": 0.0.0.0:58833 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2021-7a5zc
I0111 20:22:09.438304 140660168441856 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2021-7a5zc
network: global/networks/default-vpc
portRange: 58833
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2021-7a5zc
...
I0111 20:22:10.330671 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004529440-60eb14d51563f-d48da57a-1f010279
I0111 20:22:42.813504 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0111 20:22:42.813761 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0111 20:22:42.813870 140660168441856 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2021-7a5zc: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0111 20:22:42.814109 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2021-7a5zc%22;timeRange=2024-01-11T20:22:42.813951Z%2F2024-01-11T20:52:42.813951Z?project=grpc-testing
I0111 20:22:42.819211 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2021-7a5zc
labels:
name: psm-interop-server-20240111-2021-7a5zc
owner: xds-k8s-interop-test
...
I0111 20:22:42.972466 140660168441856 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2021-7a5zc created
I0111 20:22:42.975427 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2021-7a5zc-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:22:43.046399 140660168441856 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:22:43.046630 140660168441856 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2021-7a5zc
I0111 20:22:43.145320 140660168441856 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2021-7a5zc: neg_name=psm-interop-server-20240111-2021-7a5zc-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:22:43.145573 140660168441856 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2021-7a5zc/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:22:43.860912 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:22:43.919785 140660168441856 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:22:43.928897 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2021-7a5zc-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:22:44.003890 140660168441856 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:22:44.004150 140660168441856 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0111 20:22:44.186730 140660168441856 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:22:44.187257 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-bp577 to start
I0111 20:22:47.496472 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-bp577 ready, IP: 10.120.4.250
I0111 20:22:47.496746 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-bp577 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-bp577.log
I0111 20:22:47.497256 140660075652672 k8s_log_collector.py:70] Starting log collection thread 140660075652672 for psm-grpc-server-6dc84f789b-bp577
I0111 20:22:47.497416 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-cb857 to start
I0111 20:22:47.676731 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-cb857 ready, IP: 10.120.3.106
I0111 20:22:47.677074 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-cb857 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-cb857.log
I0111 20:22:47.677571 140660067259968 k8s_log_collector.py:70] Starting log collection thread 140660067259968 for psm-grpc-server-6dc84f789b-cb857
I0111 20:22:47.677767 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-gn4zb to start
I0111 20:22:47.779759 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-gn4zb ready, IP: 10.120.2.176
I0111 20:22:47.780034 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-gn4zb to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-gn4zb.log
I0111 20:22:47.780475 140660058867264 k8s_log_collector.py:70] Starting log collection thread 140660058867264 for psm-grpc-server-6dc84f789b-gn4zb
I0111 20:22:47.780663 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-jz7n8 to start
I0111 20:22:47.959130 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-jz7n8 ready, IP: 10.120.0.73
I0111 20:22:47.959456 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-jz7n8 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-jz7n8.log
I0111 20:22:47.959986 140659843397184 k8s_log_collector.py:70] Starting log collection thread 140659843397184 for psm-grpc-server-6dc84f789b-jz7n8
I0111 20:22:47.960211 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-kpttb to start
I0111 20:22:48.072962 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-kpttb ready, IP: 10.120.8.65
I0111 20:22:48.073245 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-kpttb to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-kpttb.log
I0111 20:22:48.073747 140659835004480 k8s_log_collector.py:70] Starting log collection thread 140659835004480 for psm-grpc-server-6dc84f789b-kpttb
I0111 20:22:48.074020 140660168441856 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0111 20:22:56.495351 140660168441856 k8s_base_runner.py:788] Deployment psm-grpc-server has 5 replicas available
I0111 20:22:56.496175 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0111 20:22:56.496340 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0111 20:22:56.533211 140660168441856 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:22:56.638636 140660168441856 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2021-7a5zc-psm-grpc-server" in zone us-central1-a
I0111 20:22:56.638876 140660168441856 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2021-7a5zc: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2021-7a5zc-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2021-7a5zc-psm-grpc-server', zone='us-central1-a')}
I0111 20:22:56.643366 140660168441856 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2021-7a5zc-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:22:57.738445 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004576647-60eb15021a77d-bc0d602d-0ffdd92b
I0111 20:23:43.134381 140660168441856 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2021-7a5zc to report all backends healthy: ['psm-interop-server-20240111-2021-7a5zc-psm-grpc-server']
I0111 20:24:09.021908 140660168441856 compute.py:494] Backend psm-interop-server-20240111-2021-7a5zc-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:24:13.026675 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0111 20:24:13.026959 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0111 20:24:13.027091 140660168441856 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2021-7a5zc: server_target=xds:///psm-grpc-server:58833 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0111 20:24:13.027304 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2021-7a5zc%22;timeRange=2024-01-11T20:24:13.027174Z%2F2024-01-11T20:54:13.027174Z?project=grpc-testing
I0111 20:24:13.029269 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2021-7a5zc
labels:
name: psm-interop-client-20240111-2021-7a5zc
owner: xds-k8s-interop-test
...
I0111 20:24:13.070690 140660168441856 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2021-7a5zc created
I0111 20:24:13.070942 140660168441856 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2021-7a5zc/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:24:13.682441 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:24:13.744079 140660168441856 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:24:13.752298 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2021-7a5zc
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:58833"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:24:13.826289 140660168441856 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:24:13.826526 140660168441856 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:24:13.938238 140660168441856 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-75ddf569df-nkdrp']
I0111 20:24:13.938524 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-75ddf569df-nkdrp to start
I0111 20:24:17.182074 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-client-75ddf569df-nkdrp ready, IP: 10.120.5.4
I0111 20:24:17.182358 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-75ddf569df-nkdrp to outlier_detection_test/test_app_logs/psm-interop-client-20240111-2021-7a5zc_psm-grpc-client-75ddf569df-nkdrp.log
I0111 20:24:17.182831 140659826611776 k8s_log_collector.py:70] Starting log collection thread 140659826611776 for psm-grpc-client-75ddf569df-nkdrp
I0111 20:24:17.183141 140660168441856 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:24:18.398954 140660168441856 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:24:18.399569 140660168441856 client_app.py:284] [psm-grpc-client-75ddf569df-nkdrp] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:24:18.502339 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:28.546621 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:38.591340 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:48.635727 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:58.680468 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:14.729544 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:39.788998 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:41.823925 140660168441856 client_app.py:323] [psm-grpc-client-75ddf569df-nkdrp] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:25:41.824228 140660168441856 client_app.py:294] [psm-grpc-client-75ddf569df-nkdrp] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:25:41.824503 140660168441856 client_app.py:244] [psm-grpc-client-75ddf569df-nkdrp] Waiting to report a READY channel to xds:///psm-grpc-server:58833
I0111 20:25:41.856519 140660168441856 client_app.py:371] [psm-grpc-client-75ddf569df-nkdrp] Server channel:
I0111 20:25:41.888895 140660168441856 client_app.py:384] [psm-grpc-client-75ddf569df-nkdrp] Found subchannel in state READY:
I0111 20:25:41.889298 140660168441856 client_app.py:255] [psm-grpc-client-75ddf569df-nkdrp] Channel to xds:///psm-grpc-server:58833 transitioned to state READY:
I0111 20:25:41.889430 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0111 20:25:41.889507 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0111 20:25:41.889702 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:25:41.924002 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0111 20:25:41.924207 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0111 20:25:41.924525 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:41.924742 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:42.958028 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
I0111 20:25:42.958505 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0111 20:25:42.958621 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0111 20:25:42.958865 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-6dc84f789b-bp577 error-code-2'}]}), timeout=5, wait_for_ready=True)
I0111 20:25:42.991240 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:42.991556 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:44.018147 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 10
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 22
psm-grpc-server-6dc84f789b-cb857: 23
psm-grpc-server-6dc84f789b-jz7n8: 22
psm-grpc-server-6dc84f789b-gn4zb: 23
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 22
psm-grpc-server-6dc84f789b-cb857: 23
psm-grpc-server-6dc84f789b-jz7n8: 22
psm-grpc-server-6dc84f789b-gn4zb: 23
I0111 20:25:44.018746 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 10 not less than or equal to 0 : Expected all RPCs to succeed: 10 of 100 failed.
I0111 20:25:45.020087 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:45.020440 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:46.047940 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:46.048268 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0111 20:25:46.048366 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0111 20:25:46.048505 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), timeout=5, wait_for_ready=True)
I0111 20:25:46.080642 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:46.080999 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:47.108057 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:47.108638 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:48.110038 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:48.110416 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:49.140868 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:49.141255 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:50.142601 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:50.142951 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:51.178625 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:51.179062 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:52.180418 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:52.180761 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:53.208257 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:53.208636 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:54.209966 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:54.210358 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:55.238577 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:55.238999 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:56.240319 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:56.240656 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:57.267826 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:57.268180 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:58.269450 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:58.269842 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:59.297893 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:59.298295 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:00.299602 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:00.299935 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:01.328274 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:01.328645 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:02.329962 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:02.330323 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:03.357577 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:03.358020 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:04.359321 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:04.359689 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:05.388402 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:05.388811 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:06.390127 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:06.390504 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:07.418077 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:07.418446 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:08.419702 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:08.420060 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:09.447746 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:09.448133 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:10.449466 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:10.449845 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:11.477724 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:11.478088 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:12.479361 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:12.479707 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:13.508231 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:13.508622 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:14.509884 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:14.510233 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:15.538199 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
I0111 20:26:15.538533 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0111 20:26:15.538653 140660168441856 xds_k8s_testcase.py:687] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0111 20:26:15.836634 140660168441856 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2021-7a5zc"
I0111 20:26:16.699855 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004775840-60eb15c011beb-5772243e-ee9f751c
I0111 20:26:44.995255 140660168441856 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2021-7a5zc"
I0111 20:26:45.087317 140660168441856 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2021-7a5zc"
I0111 20:26:45.515568 140660168441856 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2021-7a5zc"
I0111 20:26:45.966248 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004805518-60eb15dc5f5ac-9711354f-e902c5e5
I0111 20:26:48.230611 140660168441856 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2021-7a5zc"
I0111 20:26:48.587856 140660168441856 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2021-7a5zc"
I0111 20:26:49.325899 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004808593-60eb15df4ded8-5e8b0943-4a79d570
I0111 20:26:51.643544 140660168441856 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2021-7a5zc"
I0111 20:26:52.082852 140660168441856 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2021-7a5zc"
I0111 20:26:52.713625 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004812088-60eb15e2a344e-35ee3012-deb1a827
I0111 20:27:18.762315 140660168441856 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2021-7a5zc"
I0111 20:27:19.193530 140660168441856 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2021-7a5zc"
I0111 20:27:19.590487 140660168441856 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2021-7a5zc"
I0111 20:27:20.057117 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004839673-60eb15fcf1df8-4f1e7131-998b4452
I0111 20:27:24.541139 140660168441856 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:27:34.734542 140660168441856 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:27:34.734833 140660168441856 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2021-7a5zc/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:27:35.280054 140660168441856 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:27:35.407298 140660168441856 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:27:35.407573 140660168441856 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2021-7a5zc
I0111 20:28:05.610698 140660168441856 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2021-7a5zc deleted
I0111 20:28:05.611009 140660168441856 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:28:15.930924 140660168441856 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:28:15.931241 140660168441856 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:28:16.067694 140660168441856 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:28:16.067929 140660168441856 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2021-7a5zc/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:28:16.714301 140660168441856 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:28:16.845678 140660168441856 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:28:16.845938 140660168441856 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2021-7a5zc
I0111 20:29:18.149069 140660168441856 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2021-7a5zc deleted
I0111 20:29:18.149410 140660168441856 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:29:18.149624 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-vv7smh7f:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2021-7a5zc%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-vv7smh7f%22;timeRange=2024-01-11T20:24:13.027167Z%2F2024-01-11T20:28:05.610976Z?project=grpc-testing
I0111 20:29:18.149745 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-13xkr660:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2021-7a5zc%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-13xkr660%22;timeRange=2024-01-11T20:22:42.813943Z%2F2024-01-11T20:29:18.149251Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
I0111 20:29:18.149974 140660168441856 base_testcase.py:52] ----- TestCase __main__.OutlierDetectionTest.test_outlier_detection PASSED -----
----------------------------------------------------------------------
Ran 1 test in 488.784s
OK
+ for test in "${test_suites[@]}"
+ run_test remove_neg_test
+ local test_name=remove_neg_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test'
+ set -x
+ python -m tests.remove_neg_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:29:19.523287 140107089178624 xds_k8s_testcase.py:157] ----- Testing RemoveNegTest -----
I0111 20:29:19.523460 140107089178624 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:29:19.523785 140107089178624 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:29:19.540632 140107089178624 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:29:19.558647 140107089178624 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0111 20:29:19.563686 140107089178624 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2029-ibypu
I0111 20:29:19.857897 140107089178624 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
I0111 20:29:20.315825 140107089178624 xds_k8s_testcase.py:663] Found unused xds port: 18730
I0111 20:29:20.318073 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0111 20:29:20.318210 140107089178624 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2029-ibypu"
I0111 20:29:20.321232 140107089178624 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2029-ibypu
type: GRPC
...
I0111 20:29:20.974682 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004960381-60eb16700fb0b-ae6a0324-1a79d5cb
I0111 20:29:23.172715 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0111 20:29:23.172899 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0111 20:29:23.172973 140107089178624 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2029-ibypu"
I0111 20:29:23.181327 140107089178624 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2029-ibypu
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2029-ibypu
protocol: GRPC
...
I0111 20:29:23.946090 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004963183-60eb1672bbb22-22d40deb-676e2a27
I0111 20:29:41.212660 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0111 20:29:41.212850 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0111 20:29:41.212925 140107089178624 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2029-ibypu": psm-grpc-server:18730 -> psm-interop-backend-service-20240111-2029-ibypu
I0111 20:29:41.224841 140107089178624 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2029-ibypu
hostRules:
- hosts:
- psm-grpc-server:18730
pathMatcher: psm-interop-path-matcher-20240111-2029-ibypu
name: psm-interop-url-map-20240111-2029-ibypu
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2029-ibypu
name: psm-interop-path-matcher-20240111-2029-ibypu
...
I0111 20:29:41.876422 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004981226-60eb1683f0d53-0ab8d988-f946355a
I0111 20:29:44.069377 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0111 20:29:44.069590 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0111 20:29:44.069682 140107089178624 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2029-ibypu proxy "GRPC" to URL map psm-interop-url-map-20240111-2029-ibypu
I0111 20:29:44.071632 140107089178624 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2029-ibypu
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2029-ibypu
validate_for_proxyless: true
...
I0111 20:29:44.545775 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004984074-60eb1686a7f62-c0d6201b-a92cb6b2
I0111 20:29:46.914426 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0111 20:29:46.914657 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0111 20:29:46.914758 140107089178624 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2029-ibypu" in network "default-vpc": 0.0.0.0:18730 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2029-ibypu
I0111 20:29:46.917364 140107089178624 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2029-ibypu
network: global/networks/default-vpc
portRange: 18730
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2029-ibypu
...
I0111 20:29:47.774327 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004986919-60eb16895e9d1-9af049f3-e9cd2977
I0111 20:30:16.244135 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0111 20:30:16.244380 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0111 20:30:16.244487 140107089178624 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2029-ibypu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:30:16.244748 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22;timeRange=2024-01-11T20:30:16.244584Z%2F2024-01-11T21:00:16.244584Z?project=grpc-testing
I0111 20:30:16.249692 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2029-ibypu
labels:
name: psm-interop-server-20240111-2029-ibypu
owner: xds-k8s-interop-test
...
I0111 20:30:16.385384 140107089178624 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2029-ibypu created
I0111 20:30:16.388290 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2029-ibypu-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:30:16.459930 140107089178624 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:30:16.460145 140107089178624 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2029-ibypu
I0111 20:30:16.555035 140107089178624 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2029-ibypu: neg_name=psm-interop-server-20240111-2029-ibypu-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:30:16.555249 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:30:17.304565 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:30:17.363783 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:30:17.373074 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2029-ibypu-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:30:17.449811 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:30:17.450053 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:30:17.556433 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-66d9df9454-4tw54']
I0111 20:30:17.556758 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-66d9df9454-4tw54 to start
I0111 20:30:20.802196 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-server-66d9df9454-4tw54 ready, IP: 10.120.3.107
I0111 20:30:20.802464 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-66d9df9454-4tw54 to remove_neg_test/test_app_logs/psm-interop-server-20240111-2029-ibypu_psm-grpc-server-66d9df9454-4tw54.log
I0111 20:30:20.802940 140106995852864 k8s_log_collector.py:70] Starting log collection thread 140106995852864 for psm-grpc-server-66d9df9454-4tw54
I0111 20:30:20.803129 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:30:27.226154 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:30:27.226575 140107089178624 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-2029-ibypu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:30:27.226805 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22;timeRange=2024-01-11T20:30:27.226676Z%2F2024-01-11T21:00:27.226676Z?project=grpc-testing
I0111 20:30:27.226887 140107089178624 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240111-2029-ibypu
I0111 20:30:27.281032 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:30:27.328138 140107089178624 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 20:30:27.328392 140107089178624 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-2029-ibypu
I0111 20:30:27.422993 140107089178624 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-2029-ibypu: neg_name=psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0111 20:30:27.423218 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:30:28.115475 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:30:28.155731 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 20:30:28.164073 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:30:28.217071 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 20:30:28.217321 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 20:30:28.300264 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-796b4677-fhv6m']
I0111 20:30:28.300539 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-796b4677-fhv6m to start
I0111 20:30:31.604361 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-server-alt-796b4677-fhv6m ready, IP: 10.120.2.177
I0111 20:30:31.604636 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-796b4677-fhv6m to remove_neg_test/test_app_logs/psm-interop-server-20240111-2029-ibypu_psm-grpc-server-alt-796b4677-fhv6m.log
I0111 20:30:31.605150 140106987460160 k8s_log_collector.py:70] Starting log collection thread 140106987460160 for psm-grpc-server-alt-796b4677-fhv6m
I0111 20:30:31.605299 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 20:30:39.053493 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 20:30:39.053934 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0111 20:30:39.054048 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0111 20:30:39.090837 140107089178624 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:30:39.282478 140107089178624 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server" in zone us-central1-a
I0111 20:30:39.282725 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:30:39.287132 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:30:40.341223 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005039289-60eb16bb50369-390b13e2-633f668b
I0111 20:31:21.174078 140107089178624 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2029-ibypu to report all backends healthy: ['psm-interop-server-20240111-2029-ibypu-psm-grpc-server']
I0111 20:31:29.673355 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:31:33.757882 140107089178624 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:31:33.889382 140107089178624 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt" in zone us-central1-a
I0111 20:31:33.889635 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:31:33.894786 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:31:34.809871 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005093897-60eb16ef64458-622e0d5a-51aacb32
I0111 20:32:19.804207 140107089178624 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2029-ibypu to report all backends healthy: ['psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', 'psm-interop-server-20240111-2029-ibypu-psm-grpc-server']
I0111 20:32:20.168598 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:32:24.329395 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt in zone us-central1-a reported healthy
I0111 20:32:28.334081 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0111 20:32:28.334320 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0111 20:32:28.334432 140107089178624 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2029-ibypu: server_target=xds:///psm-grpc-server:18730 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:32:28.334674 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2029-ibypu%22;timeRange=2024-01-11T20:32:28.334511Z%2F2024-01-11T21:02:28.334511Z?project=grpc-testing
I0111 20:32:28.336619 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2029-ibypu
labels:
name: psm-interop-client-20240111-2029-ibypu
owner: xds-k8s-interop-test
...
I0111 20:32:28.399598 140107089178624 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2029-ibypu created
I0111 20:32:28.399821 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2029-ibypu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:32:29.090326 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:32:29.156451 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:32:29.165004 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2029-ibypu
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:18730"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:32:29.240021 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:32:29.240262 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:32:29.343608 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-85c5579db6-mmtfr']
I0111 20:32:29.343871 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-85c5579db6-mmtfr to start
I0111 20:32:31.534774 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-client-85c5579db6-mmtfr ready, IP: 10.120.0.75
I0111 20:32:31.535062 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-85c5579db6-mmtfr to remove_neg_test/test_app_logs/psm-interop-client-20240111-2029-ibypu_psm-grpc-client-85c5579db6-mmtfr.log
I0111 20:32:31.535550 140106979067456 k8s_log_collector.py:70] Starting log collection thread 140106979067456 for psm-grpc-client-85c5579db6-mmtfr
I0111 20:32:31.535698 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:32:33.718744 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:32:33.719341 140107089178624 client_app.py:284] [psm-grpc-client-85c5579db6-mmtfr] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:32:33.823825 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:32:43.868789 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:32:53.912834 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:03.957633 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:14.002755 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:30.051846 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:55.110978 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:57.145411 140107089178624 client_app.py:323] [psm-grpc-client-85c5579db6-mmtfr] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:33:57.145658 140107089178624 client_app.py:294] [psm-grpc-client-85c5579db6-mmtfr] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:33:57.145902 140107089178624 client_app.py:244] [psm-grpc-client-85c5579db6-mmtfr] Waiting to report a READY channel to xds:///psm-grpc-server:18730
I0111 20:33:57.177912 140107089178624 client_app.py:371] [psm-grpc-client-85c5579db6-mmtfr] Server channel:
I0111 20:33:57.210292 140107089178624 client_app.py:384] [psm-grpc-client-85c5579db6-mmtfr] Found subchannel in state READY:
I0111 20:33:57.210682 140107089178624 client_app.py:255] [psm-grpc-client-85c5579db6-mmtfr] Channel to xds:///psm-grpc-server:18730 transitioned to state READY:
I0111 20:33:57.210834 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0111 20:33:57.210911 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0111 20:33:57.211120 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:33:57.245143 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0111 20:33:57.245348 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0111 20:33:57.245553 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:34:01.252782 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-796b4677-fhv6m: 88
psm-grpc-server-66d9df9454-4tw54: 12
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-796b4677-fhv6m: 88
psm-grpc-server-66d9df9454-4tw54: 12
I0111 20:34:01.253057 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0111 20:34:01.253152 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0111 20:34:01.253484 140107089178624 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-66d9df9454-4tw54', 'psm-grpc-server-alt-796b4677-fhv6m']
I0111 20:34:01.253638 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:34:05.252764 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-796b4677-fhv6m: 48
psm-grpc-server-66d9df9454-4tw54: 52
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-796b4677-fhv6m: 48
psm-grpc-server-66d9df9454-4tw54: 52
I0111 20:34:05.403217 140107089178624 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:34:05.703940 140107089178624 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt" in zone us-central1-a
I0111 20:34:05.704170 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:34:05.708787 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:34:06.595280 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005245710-60eb17802c23e-fb9a4aad-eb01346e
I0111 20:34:56.023252 140107089178624 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-66d9df9454-4tw54']
I0111 20:34:56.023592 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:35:00.052741 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-66d9df9454-4tw54: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-66d9df9454-4tw54: 100
I0111 20:35:00.053199 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0111 20:35:00.053337 140107089178624 xds_k8s_testcase.py:687] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0111 20:35:00.191177 140107089178624 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2029-ibypu"
I0111 20:35:01.072874 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005300194-60eb17b421d70-d25d64e6-ad7e3c2c
I0111 20:35:29.158085 140107089178624 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2029-ibypu"
I0111 20:35:29.268464 140107089178624 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2029-ibypu"
I0111 20:35:29.691504 140107089178624 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2029-ibypu"
I0111 20:35:30.153193 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005329694-60eb17d043e8c-6b87a1d9-8c4083ad
I0111 20:35:32.544122 140107089178624 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2029-ibypu"
I0111 20:35:33.008916 140107089178624 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2029-ibypu"
I0111 20:35:33.734484 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005333014-60eb17d36e681-938d7eca-7da06934
I0111 20:35:36.074776 140107089178624 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2029-ibypu"
I0111 20:35:36.518625 140107089178624 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2029-ibypu"
I0111 20:35:37.278059 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005336524-60eb17d6c753b-00840ed1-5d890841
I0111 20:36:03.198721 140107089178624 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2029-ibypu"
I0111 20:36:03.714750 140107089178624 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2029-ibypu"
I0111 20:36:04.186576 140107089178624 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2029-ibypu"
I0111 20:36:04.817249 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005364282-60eb17f140499-5f2cd608-bd93e639
I0111 20:36:09.306283 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:36:19.513892 140107089178624 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:36:19.514219 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2029-ibypu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:36:20.077144 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:36:20.194626 140107089178624 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:36:20.194863 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2029-ibypu
I0111 20:36:50.378154 140107089178624 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2029-ibypu deleted
I0111 20:36:50.378445 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:37:00.571100 140107089178624 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:37:00.571406 140107089178624 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:37:00.689821 140107089178624 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:37:00.690067 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:37:01.520982 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:37:01.639441 140107089178624 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:37:01.639869 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2029-ibypu
I0111 20:38:01.886046 140107089178624 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2029-ibypu deleted
I0111 20:38:01.886358 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:38:01.944930 140107089178624 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:38:01.997323 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:38:02.567894 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:38:02.624440 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2029-ibypu
I0111 20:38:02.660602 140107089178624 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:38:02.660977 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-h74zgxc4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2029-ibypu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-h74zgxc4%22;timeRange=2024-01-11T20:32:28.334503Z%2F2024-01-11T20:36:50.378410Z?project=grpc-testing
I0111 20:38:02.661146 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-jydquvhe:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-jydquvhe%22;timeRange=2024-01-11T20:30:16.244575Z%2F2024-01-11T20:38:01.886324Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
I0111 20:38:02.661428 140107089178624 base_testcase.py:52] ----- TestCase __main__.RemoveNegTest.test_remove_neg PASSED -----
----------------------------------------------------------------------
Ran 1 test in 523.141s
OK
+ for test in "${test_suites[@]}"
+ run_test round_robin_test
+ local test_name=round_robin_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test'
+ set -x
+ python -m tests.round_robin_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:38:04.018293 140220474249216 xds_k8s_testcase.py:157] ----- Testing RoundRobinTest -----
I0111 20:38:04.018457 140220474249216 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:38:04.018697 140220474249216 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:38:04.035683 140220474249216 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:38:04.052942 140220474249216 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0111 20:38:04.057728 140220474249216 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2038-o77hx
I0111 20:38:04.355690 140220474249216 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
I0111 20:38:04.728399 140220474249216 xds_k8s_testcase.py:663] Found unused xds port: 5405
I0111 20:38:04.728780 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0111 20:38:04.728887 140220474249216 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2038-o77hx"
I0111 20:38:04.732052 140220474249216 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2038-o77hx
type: GRPC
...
I0111 20:38:05.473416 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005484797-60eb18642edc1-c9bfd312-264182ec
I0111 20:38:07.856784 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0111 20:38:07.857042 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0111 20:38:07.857143 140220474249216 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2038-o77hx"
I0111 20:38:07.865239 140220474249216 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2038-o77hx
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2038-o77hx
protocol: GRPC
...
I0111 20:38:08.674311 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005487867-60eb18671c564-9daf6266-1b239c55
I0111 20:38:23.742521 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0111 20:38:23.742748 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0111 20:38:23.742840 140220474249216 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2038-o77hx": psm-grpc-server:5405 -> psm-interop-backend-service-20240111-2038-o77hx
I0111 20:38:23.754699 140220474249216 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2038-o77hx
hostRules:
- hosts:
- psm-grpc-server:5405
pathMatcher: psm-interop-path-matcher-20240111-2038-o77hx
name: psm-interop-url-map-20240111-2038-o77hx
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2038-o77hx
name: psm-interop-path-matcher-20240111-2038-o77hx
...
I0111 20:38:24.422335 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005503757-60eb187643b87-c5ba0228-ec91a1a8
I0111 20:38:26.717602 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0111 20:38:26.717848 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0111 20:38:26.717947 140220474249216 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2038-o77hx proxy "GRPC" to URL map psm-interop-url-map-20240111-2038-o77hx
I0111 20:38:26.720094 140220474249216 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2038-o77hx
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2038-o77hx
validate_for_proxyless: true
...
I0111 20:38:27.276390 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005506722-60eb187917988-997898ee-2cd5b27e
I0111 20:38:29.666043 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0111 20:38:29.666253 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0111 20:38:29.666373 140220474249216 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2038-o77hx" in network "default-vpc": 0.0.0.0:5405 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2038-o77hx
I0111 20:38:29.668943 140220474249216 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2038-o77hx
network: global/networks/default-vpc
portRange: 5405
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2038-o77hx
...
I0111 20:38:30.442459 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005509673-60eb187be82ea-89b0d0e7-ea5f4315
I0111 20:38:51.959462 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0111 20:38:51.959719 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0111 20:38:51.959822 140220474249216 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2038-o77hx: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0111 20:38:51.960082 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2038-o77hx%22;timeRange=2024-01-11T20:38:51.959923Z%2F2024-01-11T21:08:51.959923Z?project=grpc-testing
I0111 20:38:51.964933 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2038-o77hx
labels:
name: psm-interop-server-20240111-2038-o77hx
owner: xds-k8s-interop-test
...
I0111 20:38:52.127088 140220474249216 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2038-o77hx created
I0111 20:38:52.129953 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2038-o77hx-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:38:52.202089 140220474249216 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:38:52.202331 140220474249216 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2038-o77hx
I0111 20:38:52.303960 140220474249216 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2038-o77hx: neg_name=psm-interop-server-20240111-2038-o77hx-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:38:52.304182 140220474249216 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2038-o77hx/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:38:53.019284 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:38:53.079222 140220474249216 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:38:53.087938 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2038-o77hx-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:38:53.166104 140220474249216 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:38:53.166345 140220474249216 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0111 20:38:53.322724 140220474249216 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-6cfcf655dd-lsbbq', 'psm-grpc-server-6cfcf655dd-p2nf6']
I0111 20:38:53.323058 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cfcf655dd-lsbbq to start
I0111 20:38:56.556079 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-server-6cfcf655dd-lsbbq ready, IP: 10.120.2.179
I0111 20:38:56.556338 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cfcf655dd-lsbbq to round_robin_test/test_app_logs/psm-interop-server-20240111-2038-o77hx_psm-grpc-server-6cfcf655dd-lsbbq.log
I0111 20:38:56.556788 140220380472896 k8s_log_collector.py:70] Starting log collection thread 140220380472896 for psm-grpc-server-6cfcf655dd-lsbbq
I0111 20:38:56.556936 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cfcf655dd-p2nf6 to start
I0111 20:38:56.730018 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-server-6cfcf655dd-p2nf6 ready, IP: 10.120.0.76
I0111 20:38:56.730258 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cfcf655dd-p2nf6 to round_robin_test/test_app_logs/psm-interop-server-20240111-2038-o77hx_psm-grpc-server-6cfcf655dd-p2nf6.log
I0111 20:38:56.730705 140220372080192 k8s_log_collector.py:70] Starting log collection thread 140220372080192 for psm-grpc-server-6cfcf655dd-p2nf6
I0111 20:38:56.730841 140220474249216 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0111 20:39:03.219086 140220474249216 k8s_base_runner.py:788] Deployment psm-grpc-server has 2 replicas available
I0111 20:39:03.219639 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0111 20:39:03.219774 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0111 20:39:03.256953 140220474249216 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:39:03.597664 140220474249216 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2038-o77hx-psm-grpc-server" in zone us-central1-a
I0111 20:39:03.597927 140220474249216 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2038-o77hx: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2038-o77hx-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2038-o77hx-psm-grpc-server', zone='us-central1-a')}
I0111 20:39:03.602385 140220474249216 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2038-o77hx-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:39:04.437691 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005543604-60eb189c44086-0c162087-12b3a6bf
I0111 20:39:45.558553 140220474249216 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2038-o77hx to report all backends healthy: ['psm-interop-server-20240111-2038-o77hx-psm-grpc-server']
I0111 20:39:58.389801 140220474249216 compute.py:494] Backend psm-interop-server-20240111-2038-o77hx-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:40:02.394601 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0111 20:40:02.394835 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0111 20:40:02.394946 140220474249216 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2038-o77hx: server_target=xds:///psm-grpc-server:5405 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:40:02.395180 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2038-o77hx%22;timeRange=2024-01-11T20:40:02.395048Z%2F2024-01-11T21:10:02.395048Z?project=grpc-testing
I0111 20:40:02.397136 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2038-o77hx
labels:
name: psm-interop-client-20240111-2038-o77hx
owner: xds-k8s-interop-test
...
I0111 20:40:02.439748 140220474249216 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2038-o77hx created
I0111 20:40:02.440038 140220474249216 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2038-o77hx/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:40:03.166345 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:40:03.230656 140220474249216 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:40:03.239415 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2038-o77hx
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:5405"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:40:03.315493 140220474249216 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:40:03.315780 140220474249216 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:40:03.423839 140220474249216 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5b6d5cdb69-k5n92']
I0111 20:40:03.424149 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-5b6d5cdb69-k5n92 to start
I0111 20:40:07.704100 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-client-5b6d5cdb69-k5n92 ready, IP: 10.120.3.109
I0111 20:40:07.704365 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-5b6d5cdb69-k5n92 to round_robin_test/test_app_logs/psm-interop-client-20240111-2038-o77hx_psm-grpc-client-5b6d5cdb69-k5n92.log
I0111 20:40:07.704831 140220363687488 k8s_log_collector.py:70] Starting log collection thread 140220363687488 for psm-grpc-client-5b6d5cdb69-k5n92
I0111 20:40:07.704979 140220474249216 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:40:09.883449 140220474249216 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:40:09.884044 140220474249216 client_app.py:284] [psm-grpc-client-5b6d5cdb69-k5n92] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:40:09.986282 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:20.031319 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:30.076177 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:40.120726 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:50.165464 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:06.214444 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:31.274459 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:33.309262 140220474249216 client_app.py:323] [psm-grpc-client-5b6d5cdb69-k5n92] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:41:33.309552 140220474249216 client_app.py:294] [psm-grpc-client-5b6d5cdb69-k5n92] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:41:33.309814 140220474249216 client_app.py:244] [psm-grpc-client-5b6d5cdb69-k5n92] Waiting to report a READY channel to xds:///psm-grpc-server:5405
I0111 20:41:33.342113 140220474249216 client_app.py:371] [psm-grpc-client-5b6d5cdb69-k5n92] Server channel:
I0111 20:41:33.374472 140220474249216 client_app.py:384] [psm-grpc-client-5b6d5cdb69-k5n92] Found subchannel in state READY:
I0111 20:41:33.374861 140220474249216 client_app.py:255] [psm-grpc-client-5b6d5cdb69-k5n92] Channel to xds:///psm-grpc-server:5405 transitioned to state READY:
I0111 20:41:33.375025 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0111 20:41:33.375128 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0111 20:41:33.375325 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:41:33.409441 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0111 20:41:33.409652 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0111 20:41:33.409900 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:41:37.407031 140220474249216 xds_k8s_testcase.py:601] [psm-grpc-client-5b6d5cdb69-k5n92] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
I0111 20:41:37.407300 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0111 20:41:37.407393 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0111 20:41:37.407519 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:41:41.406090 140220474249216 xds_k8s_testcase.py:601] [psm-grpc-client-5b6d5cdb69-k5n92] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
I0111 20:41:41.406372 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0111 20:41:41.406490 140220474249216 xds_k8s_testcase.py:687] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0111 20:41:41.619422 140220474249216 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2038-o77hx"
I0111 20:41:42.323336 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005701623-60eb1932f6df6-0cafd3d9-af03f35c
I0111 20:42:10.679920 140220474249216 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2038-o77hx"
I0111 20:42:10.791017 140220474249216 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2038-o77hx"
I0111 20:42:11.199696 140220474249216 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2038-o77hx"
I0111 20:42:11.797508 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005731202-60eb194f2c689-053f8795-55058474
I0111 20:42:14.096377 140220474249216 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2038-o77hx"
I0111 20:42:14.505065 140220474249216 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2038-o77hx"
I0111 20:42:15.345269 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005734510-60eb195253fb6-c287a5a9-4c6203aa
I0111 20:42:17.700789 140220474249216 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2038-o77hx"
I0111 20:42:18.068951 140220474249216 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2038-o77hx"
I0111 20:42:18.782963 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005738074-60eb1955ba023-084515bc-ff76eb75
I0111 20:42:44.716268 140220474249216 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2038-o77hx"
I0111 20:42:45.213931 140220474249216 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2038-o77hx"
I0111 20:42:45.688411 140220474249216 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2038-o77hx"
I0111 20:42:46.205750 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005765783-60eb197026f1a-1d348629-a0ac992c
I0111 20:42:50.756645 140220474249216 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:43:00.933695 140220474249216 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:43:00.934101 140220474249216 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2038-o77hx/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:43:01.533618 140220474249216 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:43:01.656221 140220474249216 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:43:01.656483 140220474249216 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2038-o77hx
I0111 20:43:31.842261 140220474249216 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2038-o77hx deleted
I0111 20:43:31.842575 140220474249216 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:43:42.058790 140220474249216 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:43:42.059175 140220474249216 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:43:42.181828 140220474249216 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:43:42.182139 140220474249216 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2038-o77hx/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:43:42.782669 140220474249216 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:43:42.898220 140220474249216 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:43:42.898451 140220474249216 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2038-o77hx
I0111 20:44:13.078806 140220474249216 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2038-o77hx deleted
I0111 20:44:13.079144 140220474249216 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:44:13.079380 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-uh9k1av0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2038-o77hx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-uh9k1av0%22;timeRange=2024-01-11T20:40:02.395040Z%2F2024-01-11T20:43:31.842538Z?project=grpc-testing
I0111 20:44:13.079509 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-nbgcxtq1:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2038-o77hx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-nbgcxtq1%22;timeRange=2024-01-11T20:38:51.959915Z%2F2024-01-11T20:44:13.078990Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
I0111 20:44:13.079764 140220474249216 base_testcase.py:52] ----- TestCase __main__.RoundRobinTest.test_round_robin PASSED -----
----------------------------------------------------------------------
Ran 1 test in 369.064s
OK
+ for test in "${test_suites[@]}"
+ run_test bootstrap_generator_test
+ local test_name=bootstrap_generator_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test'
+ set -x
+ python -m tests.bootstrap_generator_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:44:14.519476 140142540660736 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorClientTest -----
I0111 20:44:14.519663 140142540660736 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:44:14.519982 140142540660736 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:44:14.536901 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:44:14.553819 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0111 20:44:14.558431 140142540660736 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240111-2044-sv6uq
I0111 20:44:15.118297 140142540660736 bootstrap_generator_testcase.py:88] Found unused xds port: 26747
I0111 20:44:15.118662 140142540660736 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2044-sv6uq"
I0111 20:44:15.122098 140142540660736 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240111-2044-sv6uq
type: GRPC
...
I0111 20:44:15.754775 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005855182-60eb19c568f98-e50fc3f2-742257b1
I0111 20:44:18.207105 140142540660736 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2044-sv6uq"
I0111 20:44:18.215328 140142540660736 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2044-sv6uq
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2044-sv6uq
protocol: GRPC
...
I0111 20:44:19.089204 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005858218-60eb19c84e14f-481253f9-93b9b277
I0111 20:44:34.387290 140142540660736 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2044-sv6uq": psm-grpc-server:26747 -> psm-interop-backend-service-20240111-2044-sv6uq
I0111 20:44:34.398491 140142540660736 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2044-sv6uq
hostRules:
- hosts:
- psm-grpc-server:26747
pathMatcher: psm-interop-path-matcher-20240111-2044-sv6uq
name: psm-interop-url-map-20240111-2044-sv6uq
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2044-sv6uq
name: psm-interop-path-matcher-20240111-2044-sv6uq
...
I0111 20:44:35.099933 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005874400-60eb19d7bcd41-53aa7a7b-f99e4c44
I0111 20:44:37.449626 140142540660736 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2044-sv6uq proxy "GRPC" to URL map psm-interop-url-map-20240111-2044-sv6uq
I0111 20:44:37.451752 140142540660736 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2044-sv6uq
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2044-sv6uq
validate_for_proxyless: true
...
I0111 20:44:37.942819 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005877454-60eb19daa64e3-bc29cdab-0b570a0f
I0111 20:44:40.226653 140142540660736 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2044-sv6uq" in network "default-vpc": 0.0.0.0:26747 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2044-sv6uq
I0111 20:44:40.229472 140142540660736 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2044-sv6uq
network: global/networks/default-vpc
portRange: 26747
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2044-sv6uq
...
I0111 20:44:41.151144 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005880231-60eb19dd4c5d4-1c563b7c-761c367e
I0111 20:44:54.207493 140142540660736 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
I0111 20:44:54.329736 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2044-sv6uq: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:44:54.330115 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:44:54.329955Z%2F2024-01-11T21:14:54.329955Z?project=grpc-testing
I0111 20:44:54.335117 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2044-sv6uq
labels:
name: psm-interop-server-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:44:54.465403 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2044-sv6uq created
I0111 20:44:54.468483 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2044-sv6uq-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:44:54.541352 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:44:54.541559 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2044-sv6uq
I0111 20:44:54.641555 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2044-sv6uq: neg_name=psm-interop-server-20240111-2044-sv6uq-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:44:54.641776 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2044-sv6uq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:44:55.319915 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:44:55.381590 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:44:55.391147 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2044-sv6uq-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:44:55.457427 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:44:55.457651 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:44:55.560288 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6454f4b8bd-7f6bb']
I0111 20:44:55.560539 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6454f4b8bd-7f6bb to start
I0111 20:44:58.800908 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-6454f4b8bd-7f6bb ready, IP: 10.120.2.180
I0111 20:44:58.801188 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6454f4b8bd-7f6bb to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2044-sv6uq_psm-grpc-server-6454f4b8bd-7f6bb.log
I0111 20:44:58.801652 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-6454f4b8bd-7f6bb
I0111 20:44:58.801825 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:45:05.221496 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:45:05.258428 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:45:05.395655 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2044-sv6uq-psm-grpc-server" in zone us-central1-a
I0111 20:45:05.395876 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2044-sv6uq: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2044-sv6uq-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2044-sv6uq-psm-grpc-server', zone='us-central1-a')}
I0111 20:45:05.400449 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2044-sv6uq-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:45:06.353496 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005905402-60eb19f54db7f-f2326072-bf036fdd
I0111 20:45:57.965567 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2044-sv6uq to report all backends healthy: ['psm-interop-server-20240111-2044-sv6uq-psm-grpc-server']
I0111 20:46:11.082528 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2044-sv6uq-psm-grpc-server in zone us-central1-a reported healthy
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:46:15.087495 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.14.0 -----
I0111 20:46:15.089928 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:46:15.090277 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:46:15.090115Z%2F2024-01-11T21:16:15.090115Z?project=grpc-testing
I0111 20:46:15.092195 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:46:15.246905 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:46:15.247134 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:46:15.890919 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:46:15.946979 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:46:15.955327 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:46:16.029316 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:46:16.029568 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:46:16.140331 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-64bcdb54fd-f92ls']
I0111 20:46:16.140623 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-64bcdb54fd-f92ls to start
I0111 20:46:18.332383 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-64bcdb54fd-f92ls ready, IP: 10.120.0.77
I0111 20:46:18.332676 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-64bcdb54fd-f92ls to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-64bcdb54fd-f92ls.log
I0111 20:46:18.333166 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-64bcdb54fd-f92ls
I0111 20:46:18.333339 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:46:20.582031 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:46:20.582584 140142540660736 client_app.py:244] [psm-grpc-client-64bcdb54fd-f92ls] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:46:20.686862 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:30.731376 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:40.776571 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:50.821987 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:00.867419 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:16.916966 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:41.977340 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:48:07.037137 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:48:07.069717 140142540660736 client_app.py:384] [psm-grpc-client-64bcdb54fd-f92ls] Found subchannel in state READY:
I0111 20:48:07.069964 140142540660736 client_app.py:255] [psm-grpc-client-64bcdb54fd-f92ls] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:48:07.070204 140142540660736 grpc.py:75] [psm-grpc-client-64bcdb54fd-f92ls:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:48:07.104833 140142540660736 grpc.py:75] [psm-grpc-client-64bcdb54fd-f92ls:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:48:11.121255 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-64bcdb54fd-f92ls] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:48:11.121552 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0111 20:48:11.121896 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:48:21.393244 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:48:21.393556 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:48:22.027455 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:48:22.141269 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:48:22.141526 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:48:52.328513 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:48:52.329218 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 20:48:52.329458 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.13.0 -----
I0111 20:48:52.331351 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:48:52.331620 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:48:52.331473Z%2F2024-01-11T21:18:52.331473Z?project=grpc-testing
I0111 20:48:52.333432 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:48:52.395277 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:48:52.395504 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:48:52.982383 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:48:53.035788 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:48:53.043929 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:48:53.118485 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:48:53.118724 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:48:53.224979 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8586888567-s9xnk']
I0111 20:48:53.225250 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-8586888567-s9xnk to start
I0111 20:48:56.468243 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-8586888567-s9xnk ready, IP: 10.120.0.78
I0111 20:48:56.468545 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-8586888567-s9xnk to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-8586888567-s9xnk.log
I0111 20:48:56.468957 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-8586888567-s9xnk
I0111 20:48:56.469113 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:48:57.588819 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:48:57.589401 140142540660736 client_app.py:244] [psm-grpc-client-8586888567-s9xnk] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:48:57.686754 140142540660736 client_app.py:371] [psm-grpc-client-8586888567-s9xnk] Server channel:
I0111 20:48:57.718690 140142540660736 client_app.py:384] [psm-grpc-client-8586888567-s9xnk] Found subchannel in state READY:
I0111 20:48:57.719112 140142540660736 client_app.py:255] [psm-grpc-client-8586888567-s9xnk] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:48:57.719382 140142540660736 grpc.py:75] [psm-grpc-client-8586888567-s9xnk:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:48:57.752972 140142540660736 grpc.py:75] [psm-grpc-client-8586888567-s9xnk:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:49:01.784570 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-8586888567-s9xnk] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:49:01.784882 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0111 20:49:01.785272 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:49:11.936257 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:49:11.936563 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:49:12.486939 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:49:12.561225 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:49:12.561454 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:49:42.704245 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 20:49:42.704996 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 20:49:42.705246 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.12.0 -----
I0111 20:49:42.707194 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:49:42.707479 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:49:42.707362Z%2F2024-01-11T21:19:42.707362Z?project=grpc-testing
I0111 20:49:42.709454 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:49:42.751939 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:49:42.752166 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:49:43.386118 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:49:43.426364 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:49:43.434303 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:49:43.488190 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:49:43.488429 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:49:43.575676 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-58fcf9686d-czbd6']
I0111 20:49:43.575960 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-58fcf9686d-czbd6 to start
I0111 20:49:46.815373 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-58fcf9686d-czbd6 ready, IP: 10.120.3.111
I0111 20:49:46.815662 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-58fcf9686d-czbd6 to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-58fcf9686d-czbd6.log
I0111 20:49:46.816020 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-58fcf9686d-czbd6
I0111 20:49:46.816153 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:49:47.936704 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:49:47.937278 140142540660736 client_app.py:244] [psm-grpc-client-58fcf9686d-czbd6] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:49:48.037058 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:49:58.082522 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:08.127722 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:18.173596 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:28.218681 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:44.268203 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:51:09.328310 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:51:09.361263 140142540660736 client_app.py:384] [psm-grpc-client-58fcf9686d-czbd6] Found subchannel in state READY:
I0111 20:51:09.361513 140142540660736 client_app.py:255] [psm-grpc-client-58fcf9686d-czbd6] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:51:09.361730 140142540660736 grpc.py:75] [psm-grpc-client-58fcf9686d-czbd6:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:51:09.395957 140142540660736 grpc.py:75] [psm-grpc-client-58fcf9686d-czbd6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:51:13.395061 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-58fcf9686d-czbd6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:51:13.395328 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0111 20:51:13.395743 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:51:23.577870 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:51:23.578180 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:51:24.195605 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:51:24.308147 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:51:24.308376 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:51:54.489159 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 20:51:54.489754 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 20:51:54.489979 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.11.0 -----
I0111 20:51:54.491907 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:51:54.492170 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:51:54.492060Z%2F2024-01-11T21:21:54.492060Z?project=grpc-testing
I0111 20:51:54.493946 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:51:54.553177 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:51:54.553392 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:51:55.088105 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:51:55.152412 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:51:55.160494 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:51:55.229956 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:51:55.230192 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:51:55.336387 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7fd84cbb98-xkwlc']
I0111 20:51:55.336656 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7fd84cbb98-xkwlc to start
I0111 20:51:58.579613 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-7fd84cbb98-xkwlc ready, IP: 10.120.3.112
I0111 20:51:58.579880 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7fd84cbb98-xkwlc to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-7fd84cbb98-xkwlc.log
I0111 20:51:58.580292 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-7fd84cbb98-xkwlc
I0111 20:51:58.580427 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:51:59.700123 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:51:59.700702 140142540660736 client_app.py:244] [psm-grpc-client-7fd84cbb98-xkwlc] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:51:59.801590 140142540660736 client_app.py:371] [psm-grpc-client-7fd84cbb98-xkwlc] Server channel:
I0111 20:51:59.834794 140142540660736 client_app.py:384] [psm-grpc-client-7fd84cbb98-xkwlc] Found subchannel in state READY:
I0111 20:51:59.835221 140142540660736 client_app.py:255] [psm-grpc-client-7fd84cbb98-xkwlc] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:51:59.835451 140142540660736 grpc.py:75] [psm-grpc-client-7fd84cbb98-xkwlc:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:51:59.870186 140142540660736 grpc.py:75] [psm-grpc-client-7fd84cbb98-xkwlc:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:52:03.885331 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-7fd84cbb98-xkwlc] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:52:03.885614 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0111 20:52:03.885957 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:52:14.018023 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:52:14.018325 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:52:14.549336 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:52:14.624279 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:52:14.624495 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:52:44.766071 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 20:52:44.766637 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0111 20:52:44.824572 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:52:45.145194 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2044-sv6uq-psm-grpc-server" in zone us-central1-a
I0111 20:52:45.145443 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2044-sv6uq: set()
I0111 20:52:45.149304 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 20:52:45.859028 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006365151-60eb1babc104e-9e3a5209-91286722
I0111 20:53:31.095795 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:53:41.287217 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:53:41.287512 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:53:41.415049 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:53:41.415308 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2044-sv6uq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:53:42.003391 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:53:42.121703 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:53:42.121989 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2044-sv6uq
I0111 20:54:42.363481 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2044-sv6uq deleted
I0111 20:54:42.363807 140142540660736 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2044-sv6uq"
I0111 20:54:43.019310 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006482368-60eb1c1b8a8da-5f6a418c-4ebc2c37
I0111 20:54:51.783479 140142540660736 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2044-sv6uq"
I0111 20:54:51.869429 140142540660736 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2044-sv6uq"
I0111 20:54:52.257336 140142540660736 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2044-sv6uq"
I0111 20:54:52.744244 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006492260-60eb1c24f97db-16aed3a4-b1edf4f7
I0111 20:54:54.996236 140142540660736 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2044-sv6uq"
I0111 20:54:55.378980 140142540660736 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2044-sv6uq"
I0111 20:54:56.174697 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006495384-60eb1c27f4448-cb0b4d27-c1636ae8
I0111 20:55:00.636589 140142540660736 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2044-sv6uq"
I0111 20:55:01.107977 140142540660736 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2044-sv6uq"
I0111 20:55:01.814455 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006501113-60eb1c2d6ae9d-546214cb-3b3caa96
I0111 20:55:14.896846 140142540660736 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2044-sv6uq"
I0111 20:55:15.296034 140142540660736 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2044-sv6uq"
I0111 20:55:15.674203 140142540660736 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2044-sv6uq"
I0111 20:55:16.213811 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006515749-60eb1c3b60141-e83b6290-6871796f
I0111 20:55:20.710044 140142540660736 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorServerTest -----
I0111 20:55:20.710307 140142540660736 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:55:20.711016 140142540660736 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:55:20.727207 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:55:20.743434 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0111 20:55:20.744117 140142540660736 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240111-2055-avpqw
I0111 20:55:21.282765 140142540660736 bootstrap_generator_testcase.py:88] Found unused xds port: 4923
I0111 20:55:21.283052 140142540660736 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2055-avpqw"
I0111 20:55:21.286405 140142540660736 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240111-2055-avpqw
type: GRPC
...
I0111 20:55:22.129929 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006521336-60eb1c40b421b-a13c5a1e-777df0be
I0111 20:55:24.342421 140142540660736 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2055-avpqw"
I0111 20:55:24.350029 140142540660736 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2055-avpqw
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2055-avpqw
protocol: GRPC
...
I0111 20:55:25.180847 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006524352-60eb1c43946d4-6472bcdc-4583e16c
I0111 20:55:59.598564 140142540660736 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2055-avpqw": psm-grpc-server:4923 -> psm-interop-backend-service-20240111-2055-avpqw
I0111 20:55:59.609861 140142540660736 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2055-avpqw
hostRules:
- hosts:
- psm-grpc-server:4923
pathMatcher: psm-interop-path-matcher-20240111-2055-avpqw
name: psm-interop-url-map-20240111-2055-avpqw
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2055-avpqw
name: psm-interop-path-matcher-20240111-2055-avpqw
...
I0111 20:56:00.343596 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006559611-60eb1c6534c26-9954083c-727b754d
I0111 20:56:02.593755 140142540660736 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2055-avpqw proxy "GRPC" to URL map psm-interop-url-map-20240111-2055-avpqw
I0111 20:56:02.595804 140142540660736 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2055-avpqw
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2055-avpqw
validate_for_proxyless: true
...
I0111 20:56:03.276043 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006562597-60eb1c680db4c-c750e6c1-93d67d3f
I0111 20:56:05.669324 140142540660736 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2055-avpqw" in network "default-vpc": 0.0.0.0:4923 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2055-avpqw
I0111 20:56:05.672167 140142540660736 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2055-avpqw
network: global/networks/default-vpc
portRange: 4923
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2055-avpqw
...
I0111 20:56:06.412389 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006565674-60eb1c6afcc98-01c6ad64-566bc799
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:56:34.533740 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.14.0 -----
I0111 20:56:34.656663 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:56:34.657051 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T20:56:34.656889Z%2F2024-01-11T21:26:34.656889Z?project=grpc-testing
I0111 20:56:34.658848 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 20:56:34.816548 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 20:56:34.819331 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:56:34.893602 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:56:34.893861 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 20:56:34.988657 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:56:34.988928 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:56:35.675130 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:56:35.741965 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:56:35.750231 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:56:35.814664 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:56:35.814926 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:56:35.918545 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-db7cfdd4f-qm7g8']
I0111 20:56:35.918818 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-db7cfdd4f-qm7g8 to start
I0111 20:56:39.160809 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-db7cfdd4f-qm7g8 ready, IP: 10.120.2.182
I0111 20:56:39.161074 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-db7cfdd4f-qm7g8 to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-db7cfdd4f-qm7g8.log
I0111 20:56:39.161428 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-db7cfdd4f-qm7g8
I0111 20:56:39.161561 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:56:45.507456 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:56:45.544878 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:56:45.844813 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 20:56:45.845058 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 20:56:45.849550 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:56:46.680086 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006605851-60eb1c914dd12-671f4010-630a8466
I0111 20:57:31.893793 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 20:57:57.426590 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:58:01.433539 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:58:01.433985 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T20:58:01.433813Z%2F2024-01-11T21:28:01.433813Z?project=grpc-testing
I0111 20:58:01.436079 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 20:58:01.478648 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 20:58:01.478894 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:58:02.146666 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:58:02.206452 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:58:02.215576 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:58:02.285398 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:58:02.285646 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:58:02.388480 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-f4695bfc7-f85b2']
I0111 20:58:02.388740 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-f4695bfc7-f85b2 to start
I0111 20:58:05.634322 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-f4695bfc7-f85b2 ready, IP: 10.120.3.115
I0111 20:58:05.634583 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-f4695bfc7-f85b2 to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-f4695bfc7-f85b2.log
I0111 20:58:05.634940 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-f4695bfc7-f85b2
I0111 20:58:05.635087 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:58:06.773778 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:58:06.774339 140142540660736 client_app.py:244] [psm-grpc-client-f4695bfc7-f85b2] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 20:58:06.874165 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:16.920148 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:26.965694 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:37.010961 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:47.055922 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:03.105623 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:28.165620 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:53.225650 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 21:00:18.285653 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 21:00:18.318724 140142540660736 client_app.py:384] [psm-grpc-client-f4695bfc7-f85b2] Found subchannel in state READY:
I0111 21:00:18.319109 140142540660736 client_app.py:255] [psm-grpc-client-f4695bfc7-f85b2] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:00:18.319360 140142540660736 grpc.py:75] [psm-grpc-client-f4695bfc7-f85b2:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:00:18.353782 140142540660736 grpc.py:75] [psm-grpc-client-f4695bfc7-f85b2:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:00:22.381894 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-f4695bfc7-f85b2] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-db7cfdd4f-qm7g8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-db7cfdd4f-qm7g8: 100
I0111 21:00:22.382216 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0111 21:00:22.382571 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:00:32.664484 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:00:32.664852 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:00:33.316529 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:00:33.442970 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:00:33.443211 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:01:03.635258 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:01:03.701334 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:01:03.811141 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:01:03.811381 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:01:03.815282 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:01:04.777332 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006863817-60eb1d8751c57-03b79d28-4a1b2c7d
I0111 21:01:45.850589 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:01:56.047597 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:01:56.047880 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:01:56.162749 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:01:56.163047 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:01:56.787397 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:01:56.915694 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:01:56.915993 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:03:28.308688 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 21:03:28.309199 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 21:03:28.309388 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.13.0 -----
I0111 21:03:28.311281 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:03:28.311493 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:03:28.311385Z%2F2024-01-11T21:33:28.311385Z?project=grpc-testing
I0111 21:03:28.313409 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:03:28.373173 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:03:28.376740 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:03:28.441478 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:03:28.441701 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:03:28.534601 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:03:28.534817 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:03:29.195653 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:03:29.257096 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:03:29.265048 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:03:29.331502 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:03:29.331758 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:03:29.434362 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6cc98d756c-df447']
I0111 21:03:29.434633 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cc98d756c-df447 to start
I0111 21:03:32.666797 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-6cc98d756c-df447 ready, IP: 10.120.2.184
I0111 21:03:32.667063 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cc98d756c-df447 to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-6cc98d756c-df447.log
I0111 21:03:32.667413 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-6cc98d756c-df447
I0111 21:03:32.667565 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:03:40.123493 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:03:40.161025 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:03:40.435153 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:03:40.435380 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:03:40.439661 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:03:41.807591 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007020441-60eb1e1cb0125-958b35b5-2f2c006e
I0111 21:04:40.042250 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:04:40.306714 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:04:44.314125 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:04:44.314459 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:04:44.314342Z%2F2024-01-11T21:34:44.314342Z?project=grpc-testing
I0111 21:04:44.316317 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:04:44.358509 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:04:44.358735 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:04:45.004599 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:04:45.071032 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:04:45.079193 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:04:45.157081 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:04:45.157317 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:04:45.269329 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-67cb58cc7f-v69dw']
I0111 21:04:45.269598 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-67cb58cc7f-v69dw to start
I0111 21:04:48.502457 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-67cb58cc7f-v69dw ready, IP: 10.120.5.8
I0111 21:04:48.502723 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-67cb58cc7f-v69dw to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-67cb58cc7f-v69dw.log
I0111 21:04:48.503096 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-67cb58cc7f-v69dw
I0111 21:04:48.503246 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:04:48.601746 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:04:48.602331 140142540660736 client_app.py:244] [psm-grpc-client-67cb58cc7f-v69dw] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:04:48.700238 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:04:58.745362 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:08.789807 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:18.834194 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:28.878690 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:44.927716 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:09.975692 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:35.035231 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:35.067385 140142540660736 client_app.py:384] [psm-grpc-client-67cb58cc7f-v69dw] Found subchannel in state READY:
I0111 21:06:35.067677 140142540660736 client_app.py:255] [psm-grpc-client-67cb58cc7f-v69dw] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:06:35.067898 140142540660736 grpc.py:75] [psm-grpc-client-67cb58cc7f-v69dw:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:06:35.101624 140142540660736 grpc.py:75] [psm-grpc-client-67cb58cc7f-v69dw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:06:39.111793 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-67cb58cc7f-v69dw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cc98d756c-df447: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cc98d756c-df447: 100
I0111 21:06:39.112024 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0111 21:06:39.112339 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:06:49.374989 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:06:49.375309 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:06:50.144612 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:06:50.261777 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:06:50.262062 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:07:20.439661 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:07:20.507711 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:07:20.778751 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:07:20.779002 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:07:20.783113 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:07:21.691504 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007240785-60eb1eeed2edf-3dfb3df7-de009502
I0111 21:08:18.124012 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:08:28.299721 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:08:28.300086 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:08:28.398961 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:08:28.399232 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:08:28.948268 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:08:29.071103 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:08:29.071338 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:09:29.371460 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 21:09:29.372121 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 21:09:29.372370 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.12.0 -----
I0111 21:09:29.374338 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:09:29.374613 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:09:29.374488Z%2F2024-01-11T21:39:29.374488Z?project=grpc-testing
I0111 21:09:29.376492 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:09:29.435277 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:09:29.438337 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:09:29.500017 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:09:29.500236 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:09:29.595968 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:09:29.596177 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:09:30.414870 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:09:30.471748 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:09:30.479581 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:09:30.552713 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:09:30.552945 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:09:30.657209 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-796b44865d-sgnlg']
I0111 21:09:30.657461 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-796b44865d-sgnlg to start
I0111 21:09:33.895037 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-796b44865d-sgnlg ready, IP: 10.120.2.185
I0111 21:09:33.895331 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-796b44865d-sgnlg to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-796b44865d-sgnlg.log
I0111 21:09:33.895732 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-796b44865d-sgnlg
I0111 21:09:33.895989 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:09:41.286038 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:09:41.323321 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:09:41.430749 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:09:41.431002 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:09:41.435462 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:09:42.386697 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007381437-60eb1f74f5cb7-e07f8328-a494f032
I0111 21:10:38.582781 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:10:38.852439 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:10:42.859508 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:10:42.859918 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:10:42.859783Z%2F2024-01-11T21:40:42.859783Z?project=grpc-testing
I0111 21:10:42.861672 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:10:42.905064 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:10:42.905291 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:10:43.529134 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:10:43.588904 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:10:43.597135 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:10:43.667706 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:10:43.667982 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:10:43.766411 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7b7769fff5-tvbxn']
I0111 21:10:43.766671 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7b7769fff5-tvbxn to start
I0111 21:10:47.002441 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-7b7769fff5-tvbxn ready, IP: 10.120.3.118
I0111 21:10:47.002708 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7b7769fff5-tvbxn to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-7b7769fff5-tvbxn.log
I0111 21:10:47.003091 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-7b7769fff5-tvbxn
I0111 21:10:47.003233 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:10:47.097506 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:10:47.098055 140142540660736 client_app.py:244] [psm-grpc-client-7b7769fff5-tvbxn] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:10:47.197520 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:10:57.243497 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:07.288660 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:17.334598 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:27.379671 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:43.429177 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:08.489611 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:33.549543 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:58.609570 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:58.642272 140142540660736 client_app.py:384] [psm-grpc-client-7b7769fff5-tvbxn] Found subchannel in state READY:
I0111 21:12:58.642575 140142540660736 client_app.py:255] [psm-grpc-client-7b7769fff5-tvbxn] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:12:58.642813 140142540660736 grpc.py:75] [psm-grpc-client-7b7769fff5-tvbxn:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:12:58.677023 140142540660736 grpc.py:75] [psm-grpc-client-7b7769fff5-tvbxn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:13:02.697348 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-7b7769fff5-tvbxn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-796b44865d-sgnlg: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-796b44865d-sgnlg: 100
I0111 21:13:02.697618 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0111 21:13:02.697959 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:13:12.964744 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:13:12.965056 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:13:13.481553 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:13:13.595120 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:13:13.595342 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:13:43.776895 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:13:43.833095 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:13:44.208266 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:13:44.208482 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:13:44.212251 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:13:45.051813 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007624214-60eb205c7d609-01291ee3-fb94dfe6
I0111 21:14:39.532132 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:14:49.709480 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:14:49.709767 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:14:49.808210 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:14:49.808472 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:14:50.414696 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:14:50.530464 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:14:50.530708 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:16:50.949151 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 21:16:50.949785 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 21:16:50.950046 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.11.0 -----
I0111 21:16:50.952018 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:16:50.952279 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:16:50.952160Z%2F2024-01-11T21:46:50.952160Z?project=grpc-testing
I0111 21:16:50.954123 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:16:51.021661 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:16:51.024875 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:16:51.093547 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:16:51.093785 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:16:51.182155 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:16:51.182374 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:16:51.747051 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:16:51.804667 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:16:51.812747 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:16:51.892325 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:16:51.892588 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:16:52.002494 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-7657558f7-mr5vk']
I0111 21:16:52.002761 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-7657558f7-mr5vk to start
I0111 21:16:55.231371 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-7657558f7-mr5vk ready, IP: 10.120.5.10
I0111 21:16:55.231686 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-7657558f7-mr5vk to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-7657558f7-mr5vk.log
I0111 21:16:55.232069 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-7657558f7-mr5vk
I0111 21:16:55.232215 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:17:10.928242 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:17:10.964773 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:17:11.089694 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:17:11.089963 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:17:11.094450 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:17:11.895720 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007831096-60eb2121c9cf9-d131e82e-b96d96f9
I0111 21:17:46.334476 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:18:20.173809 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:18:24.180645 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:18:24.180991 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:18:24.180866Z%2F2024-01-11T21:48:24.180866Z?project=grpc-testing
I0111 21:18:24.182837 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:18:24.254984 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:18:24.255216 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:18:24.831209 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:18:24.893419 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:18:24.901731 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:18:24.970378 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:18:24.970642 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:18:25.071085 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-57b79b84f-24k5b']
I0111 21:18:25.071403 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-57b79b84f-24k5b to start
I0111 21:18:28.314275 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-57b79b84f-24k5b ready, IP: 10.120.5.11
I0111 21:18:28.314561 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-57b79b84f-24k5b to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-57b79b84f-24k5b.log
I0111 21:18:28.314958 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-57b79b84f-24k5b
I0111 21:18:28.315106 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:18:29.519833 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:18:29.520420 140142540660736 client_app.py:244] [psm-grpc-client-57b79b84f-24k5b] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:18:29.619433 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:39.665026 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:49.710551 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:59.755653 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:09.796014 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:25.840476 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:50.875952 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:50.908937 140142540660736 client_app.py:384] [psm-grpc-client-57b79b84f-24k5b] Found subchannel in state READY:
I0111 21:19:50.909192 140142540660736 client_app.py:255] [psm-grpc-client-57b79b84f-24k5b] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:19:50.909413 140142540660736 grpc.py:75] [psm-grpc-client-57b79b84f-24k5b:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:19:50.943792 140142540660736 grpc.py:75] [psm-grpc-client-57b79b84f-24k5b:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:19:54.966116 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-57b79b84f-24k5b] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7657558f7-mr5vk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7657558f7-mr5vk: 100
I0111 21:19:54.966348 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0111 21:19:54.966643 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:20:05.131322 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:20:05.131700 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:20:05.715064 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:20:05.832372 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:20:05.832652 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:20:36.068842 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:20:36.131186 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:20:36.406724 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:20:36.406948 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:20:36.410737 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:20:37.424563 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008036412-60eb21e597dd3-a2232600-2f755c95
I0111 21:21:11.934650 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:21:22.119263 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:21:22.119597 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:21:22.212309 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:21:22.212624 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:21:22.715710 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:21:22.831243 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:21:22.831511 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:23:23.257525 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 21:23:23.258229 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0111 21:23:23.258379 140142540660736 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2055-avpqw"
I0111 21:23:23.903260 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008203262-60eb2284b6a87-f5089ad9-11c70026
I0111 21:23:36.784759 140142540660736 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2055-avpqw"
I0111 21:23:36.915439 140142540660736 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2055-avpqw"
I0111 21:23:37.397751 140142540660736 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2055-avpqw"
I0111 21:23:37.970142 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008217400-60eb229232699-37a0050d-cd5b443d
I0111 21:23:40.260673 140142540660736 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2055-avpqw"
I0111 21:23:40.649794 140142540660736 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2055-avpqw"
I0111 21:23:41.324169 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008220655-60eb22954cfd6-968197e3-87811ed9
I0111 21:23:45.848365 140142540660736 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2055-avpqw"
I0111 21:23:46.279969 140142540660736 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2055-avpqw"
I0111 21:23:46.929242 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008226285-60eb229aab6e2-e874402f-f6e61c3c
I0111 21:24:00.105541 140142540660736 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2055-avpqw"
I0111 21:24:00.452727 140142540660736 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2055-avpqw"
I0111 21:24:00.862147 140142540660736 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2055-avpqw"
I0111 21:24:01.376230 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008240933-60eb22a8a3963-bdf30b90-40028f66
----------------------------------------------------------------------
Ran 8 tests in 2391.262s
OK
+ echo 'Failed test suites: 0'
Failed test suites: 0
[ID: 3109391] Command finished after 6588 secs, exit value: 0
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[13:24:06 PST] Collecting build artifacts from build VM
[13:24:07 PST] Kokoro builder finished
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[11:34:13 PST] Transferring environment variable script to build VM
[11:34:14 PST] Transferring kokoro_log_reader.py to build VM
[11:34:15 PST] Transferring source code to build VM
[11:34:17 PST] Executing build script on build VM
[ID: 3109391] Executing command via SSH:
export KOKORO_BUILD_NUMBER="697"
export KOKORO_JOB_NAME="grpc/go/master/branch/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-go/test/kokoro/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-go/test/kokoro/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/psm-interop/main/.kokoro/psm_interop_kokoro_lib.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-go
+ set +x
Kokoro Ubuntu version:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.3 LTS
Release: 22.04
Codename: jammy
Python 3.10.12
pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
Sponge properties:
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,0
GIT_ORIGIN_URL,https://github.com/zasweq/grpc-go.git
GIT_COMMIT_SHORT,bb7b7332
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 130042 files and directories currently installed.)
Removing needrestart (3.5-5ubuntu2.1) ...
Processing triggers for man-db (2.10.2-1) ...
W: https://packages.cloud.google.com/apt/dists/cloud-sdk/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
W: https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129987 files and directories currently installed.)
Removing libintl-xs-perl (1.26-3build2) ...
Removing libintl-perl (1.26-3build2) ...
Removing libmodule-find-perl (0.15-1) ...
Removing libmodule-scandeps-perl (1.31-1) ...
Removing libproc-processtable-perl:amd64 (0.634-1build1) ...
Removing libsort-naturally-perl (1.03-2) ...
Removing libterm-readkey-perl (2.38-1build4) ...
Selecting previously unselected package google-cloud-sdk-gke-gcloud-auth-plugin.
(Reading database ... (Reading database ... 5%(Reading database ... 10%(Reading database ... 15%(Reading database ... 20%(Reading database ... 25%(Reading database ... 30%(Reading database ... 35%(Reading database ... 40%(Reading database ... 45%(Reading database ... 50%(Reading database ... 55%(Reading database ... 60%(Reading database ... 65%(Reading database ... 70%(Reading database ... 75%(Reading database ... 80%(Reading database ... 85%(Reading database ... 90%(Reading database ... 95%(Reading database ... 100%(Reading database ... 129587 files and directories currently installed.)
Preparing to unpack .../google-cloud-sdk-gke-gcloud-auth-plugin_0%3a459.0.0-0_amd64.deb ...
Unpacking google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Selecting previously unselected package kubectl.
Preparing to unpack .../kubectl_1%3a459.0.0-0_amd64.deb ...
Unpacking kubectl (1:459.0.0-0) ...
Selecting previously unselected package python3-pip-whl.
Preparing to unpack .../python3-pip-whl_22.0.2+dfsg-1ubuntu0.4_all.deb ...
Unpacking python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Selecting previously unselected package python3-setuptools-whl.
Preparing to unpack .../python3-setuptools-whl_59.6.0-1.2ubuntu0.22.04.1_all.deb ...
Unpacking python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Selecting previously unselected package python3.10-venv.
Preparing to unpack .../python3.10-venv_3.10.12-1~22.04.3_amd64.deb ...
Unpacking python3.10-venv (3.10.12-1~22.04.3) ...
Setting up python3-setuptools-whl (59.6.0-1.2ubuntu0.22.04.1) ...
Setting up python3-pip-whl (22.0.2+dfsg-1ubuntu0.4) ...
Setting up google-cloud-sdk-gke-gcloud-auth-plugin (459.0.0-0) ...
Setting up kubectl (1:459.0.0-0) ...
Setting up python3.10-venv (3.10.12-1~22.04.3) ...
Processing triggers for man-db (2.10.2-1) ...
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
Cloning driver to https://github.com/grpc/psm-interop.git branch main to /tmp/tmp.mnM8lQRITP/psm-interop
Cloning into '/tmp/tmp.mnM8lQRITP/psm-interop'...
Install python dependencies
Creating python virtual environment: /tmp/tmp.mnM8lQRITP/psm-interop/venv
Collecting Mako==1.2.4 (from -r requirements.lock (line 1))
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 78.7/78.7 kB 1.8 MB/s eta 0:00:00
Collecting PyYAML==6.0 (from -r requirements.lock (line 2))
Downloading PyYAML-6.0-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (682 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 682.2/682.2 kB 7.0 MB/s eta 0:00:00
Collecting absl-py==0.15.0 (from -r requirements.lock (line 3))
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 132.0/132.0 kB 5.8 MB/s eta 0:00:00
Collecting google-api-python-client==1.12.11 (from -r requirements.lock (line 4))
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.1/62.1 kB 4.7 MB/s eta 0:00:00
Collecting google-cloud-secret-manager==2.15.1 (from -r requirements.lock (line 5))
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100.4/100.4 kB 7.0 MB/s eta 0:00:00
Collecting grpcio==1.57.0 (from -r requirements.lock (line 6))
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.0 kB)
Collecting grpcio-health-checking==1.57.0 (from -r requirements.lock (line 7))
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting grpcio-tools==1.57.0 (from -r requirements.lock (line 8))
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.2 kB)
Collecting grpcio-channelz==1.57.0 (from -r requirements.lock (line 9))
Downloading grpcio_channelz-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting kubernetes==27.2.0 (from -r requirements.lock (line 10))
Downloading kubernetes-27.2.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting six==1.16.0 (from -r requirements.lock (line 11))
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1 (from -r requirements.lock (line 12))
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==23.1 (from -r requirements.lock (line 13))
Downloading packaging-23.1-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.9/48.9 kB 5.0 MB/s eta 0:00:00
Collecting Pygments==2.14.0 (from -r requirements.lock (line 14))
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 10.9 MB/s eta 0:00:00
Collecting python-dateutil==2.8.2 (from -r requirements.lock (line 15))
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 10.6 MB/s eta 0:00:00
Collecting protobuf==4.24.1 (from -r requirements.lock (line 16))
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl.metadata (540 bytes)
Collecting xds-protos==1.58.0rc1 (from -r requirements.lock (line 17))
Downloading xds_protos-1.58.0rc1-py3-none-any.whl.metadata (1.3 kB)
Collecting cachetools==5.3.1 (from -r requirements.lock (line 19))
Downloading cachetools-5.3.1-py3-none-any.whl.metadata (5.2 kB)
Collecting certifi==2023.7.22 (from -r requirements.lock (line 20))
Downloading certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Collecting charset-normalizer==3.2.0 (from -r requirements.lock (line 21))
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)
Collecting google-api-core==2.11.1 (from -r requirements.lock (line 22))
Downloading google_api_core-2.11.1-py3-none-any.whl.metadata (2.7 kB)
Collecting google-auth==2.22.0 (from -r requirements.lock (line 23))
Downloading google_auth-2.22.0-py2.py3-none-any.whl.metadata (4.2 kB)
Collecting google-auth-httplib2==0.1.0 (from -r requirements.lock (line 24))
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.60.0 (from -r requirements.lock (line 25))
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl.metadata (1.5 kB)
Collecting grpc-google-iam-v1==0.12.6 (from -r requirements.lock (line 26))
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.57.0 (from -r requirements.lock (line 27))
Downloading grpcio_status-1.57.0-py3-none-any.whl.metadata (1.2 kB)
Collecting httplib2==0.22.0 (from -r requirements.lock (line 28))
Downloading httplib2-0.22.0-py3-none-any.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.9/96.9 kB 8.0 MB/s eta 0:00:00
Collecting idna==3.4 (from -r requirements.lock (line 29))
Downloading idna-3.4-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 5.5 MB/s eta 0:00:00
Collecting MarkupSafe==2.1.3 (from -r requirements.lock (line 30))
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.0 kB)
Collecting oauthlib==3.2.2 (from -r requirements.lock (line 31))
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 9.2 MB/s eta 0:00:00
Collecting proto-plus==1.22.3 (from -r requirements.lock (line 32))
Downloading proto_plus-1.22.3-py3-none-any.whl.metadata (2.2 kB)
Collecting pyasn1==0.5.0 (from -r requirements.lock (line 33))
Downloading pyasn1-0.5.0-py2.py3-none-any.whl (83 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 83.9/83.9 kB 8.2 MB/s eta 0:00:00
Collecting pyasn1-modules==0.3.0 (from -r requirements.lock (line 34))
Downloading pyasn1_modules-0.3.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.3/181.3 kB 11.0 MB/s eta 0:00:00
Collecting pyparsing==3.1.1 (from -r requirements.lock (line 35))
Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
Collecting requests==2.31.0 (from -r requirements.lock (line 36))
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting requests-oauthlib==1.3.1 (from -r requirements.lock (line 37))
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9 (from -r requirements.lock (line 38))
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1 (from -r requirements.lock (line 39))
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.16 (from -r requirements.lock (line 40))
Downloading urllib3-1.26.16-py2.py3-none-any.whl.metadata (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.4/48.4 kB 4.0 MB/s eta 0:00:00
Collecting websocket-client==1.6.2 (from -r requirements.lock (line 41))
Downloading websocket_client-1.6.2-py3-none-any.whl.metadata (7.5 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.10/site-packages (from grpcio-tools==1.57.0->-r requirements.lock (line 8)) (69.0.3)
Downloading grpcio-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.3 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.3/5.3 MB 21.6 MB/s eta 0:00:00
Downloading grpcio_health_checking-1.57.0-py3-none-any.whl (8.6 kB)
Downloading grpcio_tools-1.57.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.6/2.6 MB 32.5 MB/s eta 0:00:00
Downloading grpcio_channelz-1.57.0-py3-none-any.whl (10 kB)
Downloading kubernetes-27.2.0-py2.py3-none-any.whl (1.5 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 30.4 MB/s eta 0:00:00
Downloading protobuf-4.24.1-cp37-abi3-manylinux2014_x86_64.whl (311 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 311.4/311.4 kB 23.4 MB/s eta 0:00:00
Downloading xds_protos-1.58.0rc1-py3-none-any.whl (1.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 33.8 MB/s eta 0:00:00
Downloading cachetools-5.3.1-py3-none-any.whl (9.3 kB)
Downloading certifi-2023.7.22-py3-none-any.whl (158 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 158.3/158.3 kB 13.6 MB/s eta 0:00:00
Downloading charset_normalizer-3.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (201 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 201.8/201.8 kB 17.2 MB/s eta 0:00:00
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.5/120.5 kB 11.0 MB/s eta 0:00:00
Downloading google_auth-2.22.0-py2.py3-none-any.whl (181 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 181.8/181.8 kB 12.7 MB/s eta 0:00:00
Downloading googleapis_common_protos-1.60.0-py2.py3-none-any.whl (227 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 227.6/227.6 kB 19.0 MB/s eta 0:00:00
Downloading grpcio_status-1.57.0-py3-none-any.whl (5.1 kB)
Downloading MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Downloading proto_plus-1.22.3-py3-none-any.whl (48 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 48.1/48.1 kB 4.1 MB/s eta 0:00:00
Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 10.8 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 6.2 MB/s eta 0:00:00
Downloading urllib3-1.26.16-py2.py3-none-any.whl (143 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 143.1/143.1 kB 13.6 MB/s eta 0:00:00
Downloading websocket_client-1.6.2-py3-none-any.whl (57 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 57.0/57.0 kB 5.5 MB/s eta 0:00:00
Installing collected packages: websocket-client, urllib3, uritemplate, six, PyYAML, pyparsing, Pygments, pyasn1, protobuf, packaging, oauthlib, MarkupSafe, idna, grpcio, charset-normalizer, certifi, cachetools, xds-protos, tenacity, rsa, requests, python-dateutil, pyasn1-modules, proto-plus, Mako, httplib2, grpcio-tools, grpcio-health-checking, grpcio-channelz, googleapis-common-protos, absl-py, requests-oauthlib, grpcio-status, google-auth, kubernetes, grpc-google-iam-v1, google-auth-httplib2, google-api-core, google-api-python-client, google-cloud-secret-manager
Successfully installed Mako-1.2.4 MarkupSafe-2.1.3 PyYAML-6.0 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.1 certifi-2023.7.22 charset-normalizer-3.2.0 google-api-core-2.11.1 google-api-python-client-1.12.11 google-auth-2.22.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.60.0 grpc-google-iam-v1-0.12.6 grpcio-1.57.0 grpcio-channelz-1.57.0 grpcio-health-checking-1.57.0 grpcio-status-1.57.0 grpcio-tools-1.57.0 httplib2-0.22.0 idna-3.4 kubernetes-27.2.0 oauthlib-3.2.2 packaging-23.1 proto-plus-1.22.3 protobuf-4.24.1 pyasn1-0.5.0 pyasn1-modules-0.3.0 pyparsing-3.1.1 python-dateutil-2.8.2 requests-2.31.0 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.16 websocket-client-1.6.2 xds-protos-1.58.0rc1
Installed Python packages:
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.1
certifi 2023.7.22
charset-normalizer 3.2.0
google-api-core 2.11.1
google-api-python-client 1.12.11
google-auth 2.22.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.60.0
grpc-google-iam-v1 0.12.6
grpcio 1.57.0
grpcio-channelz 1.57.0
grpcio-health-checking 1.57.0
grpcio-status 1.57.0
grpcio-tools 1.57.0
httplib2 0.22.0
idna 3.4
kubernetes 27.2.0
Mako 1.2.4
MarkupSafe 2.1.3
oauthlib 3.2.2
packaging 23.1
pip 23.3.2
proto-plus 1.22.3
protobuf 4.24.1
pyasn1 0.5.0
pyasn1-modules 0.3.0
Pygments 2.14.0
pyparsing 3.1.1
python-dateutil 2.8.2
PyYAML 6.0
requests 2.31.0
requests-oauthlib 1.3.1
rsa 4.9
setuptools 69.0.3
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.16
websocket-client 1.6.2
xds-protos 1.58.0rc1
Generate python code from grpc.testing protos: protos/grpc/testing/test.proto protos/grpc/testing/messages.proto protos/grpc/testing/empty.proto
/tmp/tmp.mnM8lQRITP/psm-interop/venv/lib/python3.10/site-packages/grpc_tools/protoc.py:21: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
Generated files /tmp/tmp.mnM8lQRITP/psm-interop/protos/grpc/testing:
total 108
-rw-rw-r-- 1 kbuilder kbuilder 957 Jan 11 19:35 empty.proto
-rw-rw-r-- 1 kbuilder kbuilder 1007 Jan 11 19:35 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 11 19:35 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 11527 Jan 11 19:35 messages.proto
-rw-rw-r-- 1 kbuilder kbuilder 17206 Jan 11 19:35 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jan 11 19:35 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 4871 Jan 11 19:35 test.proto
-rw-rw-r-- 1 kbuilder kbuilder 4296 Jan 11 19:35 test_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 41870 Jan 11 19:35 test_pb2_grpc.py
Server image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
Server image not found
Client image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
Client image not found
Building Go xDS interop test app Docker images
#0 building with "default" instance using docker driver
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.51kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 2B done
#2 DONE 0.1s
#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.6s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.6s
#5 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#5 resolve docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 0.0s done
#5 sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48 1.64kB / 1.64kB done
#5 sha256:13b7e62e8df80264dbb747995705a986aa530415763a6c58f84a3ca8af9a5bcd 528B / 528B done
#5 sha256:f8c20f8bbcb684055b4fea470fdd169c86e87786940b3262335b12ec3adef418 1.47kB / 1.47kB done
#5 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.1s done
#5 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca
#5 DONE 0.5s
#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#6 resolve docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 0.0s done
#6 sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5 1.65kB / 1.65kB done
#6 sha256:2523a6f68a0f515fe251aad40b18545155135ca6a5b2e61da8254df9153e3648 1.16kB / 1.16kB done
#6 sha256:82e66f0646744e5c40c91b75ff7d5ff5577887b46d5402965633324dd842ea87 5.22kB / 5.22kB done
#6 sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 3.41MB / 3.41MB 0.1s done
#6 sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 284.20kB / 284.20kB 0.1s done
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 29.36MB / 67.09MB 0.5s
#6 sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 156B / 156B 0.2s done
#6 extracting sha256:661ff4d9561e3fd050929ee5097067c34bafc523ee60f5294a37fd08056a73ca 0.1s done
#6 extracting sha256:2979d3ffd3a49b2dd31f3e87826cb75934ebef5c0e6e44c8ae38ead06678961d 0.1s done
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 44.04MB / 67.09MB 0.6s
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 57.67MB / 67.09MB 0.7s
#6 ...
#7 [internal] load build context
#7 transferring context: 87.13MB 0.8s done
#7 DONE 0.8s
#6 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#6 sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 67.09MB / 67.09MB 1.0s done
#6 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 0.1s
#6 extracting sha256:a32e8c70f5774b3d882e17676b30b846a40001502ad8a543f3c72bebca48fcbc 4.7s done
#6 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98
#6 extracting sha256:cedd8caa912b33fa383ee6e3be5031c2bebfc4a735537ebe152941e374fb6c98 done
#6 DONE 6.1s
#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 DONE 1.2s
#9 [build 3/4] COPY . .
#9 DONE 0.5s
#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/client/client.go
#10 0.363 go: downloading golang.org/x/net v0.18.0
#10 0.377 go: downloading github.com/golang/protobuf v1.5.3
#10 0.380 go: downloading google.golang.org/protobuf v1.31.0
#10 0.394 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.406 go: downloading golang.org/x/sys v0.14.0
#10 0.439 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.468 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.838 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.845 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.893 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.340 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.340 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.343 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.343 go: downloading golang.org/x/text v0.14.0
#10 1.346 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.346 go: downloading golang.org/x/sync v0.5.0
#10 1.410 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.419 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 19.3s
#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/client .
#11 DONE 0.1s
#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:b5bd887113ca73ccd5b3949a23db4a47a846a45ad140c6f910e259aa76ce9fee done
#12 naming to gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b done
#12 DONE 0.2s
#0 building with "default" instance using docker driver
#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.51kB done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/golang:1.21-alpine
#3 DONE 0.2s
#4 [internal] load metadata for docker.io/library/alpine:latest
#4 DONE 0.2s
#5 [build 1/4] FROM docker.io/library/golang:1.21-alpine@sha256:fd78f2fb1e49bcf343079bbbb851c936a18fc694df993cbddaa24ace0cc724c5
#5 DONE 0.0s
#6 [stage-1 1/2] FROM docker.io/library/alpine@sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
#6 CACHED
#7 [internal] load build context
#7 transferring context: 76.35kB 0.1s done
#7 DONE 0.1s
#8 [build 2/4] WORKDIR /go/src/grpc-go
#8 CACHED
#9 [build 3/4] COPY . .
#9 CACHED
#10 [build 4/4] RUN go build -tags osusergo,netgo interop/xds/server/server.go
#10 0.319 go: downloading github.com/envoyproxy/go-control-plane v0.11.1
#10 0.334 go: downloading google.golang.org/protobuf v1.31.0
#10 0.335 go: downloading github.com/golang/protobuf v1.5.3
#10 0.336 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20231106174013-bbf56f31fb17
#10 0.356 go: downloading golang.org/x/sys v0.14.0
#10 0.391 go: downloading golang.org/x/net v0.18.0
#10 0.394 go: downloading google.golang.org/genproto v0.0.0-20231106174013-bbf56f31fb17
#10 0.755 go: downloading github.com/cncf/xds/go v0.0.0-20231109132714-523115ebc101
#10 0.823 go: downloading github.com/cespare/xxhash/v2 v2.2.0
#10 0.853 go: downloading github.com/cncf/udpa/go v0.0.0-20220112060539-c52dc94e7fbe
#10 1.313 go: downloading github.com/envoyproxy/protoc-gen-validate v1.0.2
#10 1.313 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20231106174013-bbf56f31fb17
#10 1.319 go: downloading golang.org/x/text v0.14.0
#10 1.322 go: downloading golang.org/x/oauth2 v0.14.0
#10 1.326 go: downloading github.com/census-instrumentation/opencensus-proto v0.4.1
#10 1.336 go: downloading golang.org/x/sync v0.5.0
#10 1.393 go: downloading cloud.google.com/go/compute/metadata v0.2.3
#10 1.406 go: downloading cloud.google.com/go/compute v1.23.3
#10 DONE 19.4s
#11 [stage-1 2/2] COPY --from=build /go/src/grpc-go/server .
#11 DONE 0.1s
#12 exporting to image
#12 exporting layers
#12 exporting layers 0.2s done
#12 writing image sha256:ace14729962dac25c084bafd8f0e2175a07a1a563e7384aecb5b447cba7048bf done
#12 naming to gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b done
#12 DONE 0.2s
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-client]
5cf64c40fd39: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
5cf64c40fd39: Pushed
bb7b7332b803301b57f6497415512a9f5471fa0b: digest: sha256:d174949a1da645bbdef7d5ece56b02f3fe5a12476216ef2b817fd18c16d8a960 size: 740
The push refers to repository [gcr.io/grpc-testing/xds-interop/go-server]
071b551139b6: Preparing
5af4f8f59b76: Preparing
5af4f8f59b76: Layer already exists
071b551139b6: Pushed
bb7b7332b803301b57f6497415512a9f5471fa0b: digest: sha256:db9bd17b6781e905324dba79d831ff007c7932068f2904eae7653ffabd59b5ff size: 740
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test'
+ python -m tests.affinity_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:36:34.201105 139848621117440 xds_k8s_testcase.py:157] ----- Testing AffinityTest -----
I0111 19:36:34.201264 139848621117440 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:36:34.201531 139848621117440 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:36:34.218817 139848621117440 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:36:34.375864 139848621117440 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] AffinityTest.test_affinity
I0111 19:36:34.652938 139848621117440 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1936-cyja4
I0111 19:36:34.948297 139848621117440 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/affinity_test
I0111 19:36:35.343053 139848621117440 xds_k8s_testcase.py:663] Found unused xds port: 41089
I0111 19:36:35.343407 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0111 19:36:35.343583 139848621117440 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1936-cyja4"
I0111 19:36:35.346664 139848621117440 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1936-cyja4
type: GRPC
...
I0111 19:36:36.106127 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001795411-60eb0aa5b5d3c-b1fa0bd1-714988f4
I0111 19:36:38.401229 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.00_create_health_check ---
I0111 19:36:38.401441 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0111 19:36:38.401545 139848621117440 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1936-cyja4"
I0111 19:36:38.409992 139848621117440 compute.py:537] Creating compute resource:
---
consistentHash:
httpHeaderName: xds_md
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1936-cyja4
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicy: RING_HASH
name: psm-interop-backend-service-20240111-1936-cyja4
protocol: GRPC
sessionAffinity: HEADER_FIELD
...
I0111 19:36:39.205381 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001798412-60eb0aa8927ef-483612cf-c38a30e8
I0111 19:37:35.390896 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.01_create_backend_services ---
I0111 19:37:35.391151 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0111 19:37:35.391245 139848621117440 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1936-cyja4": psm-grpc-server:41089 -> psm-interop-backend-service-20240111-1936-cyja4
I0111 19:37:35.402735 139848621117440 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4
hostRules:
- hosts:
- psm-grpc-server:41089
pathMatcher: psm-interop-path-matcher-20240111-1936-cyja4
name: psm-interop-url-map-20240111-1936-cyja4
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4
name: psm-interop-path-matcher-20240111-1936-cyja4
...
I0111 19:37:35.989221 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001855405-60eb0adeecbf8-445debde-cdf8b08e
I0111 19:37:38.343500 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.02_create_url_map ---
I0111 19:37:38.343738 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0111 19:37:38.343828 139848621117440 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1936-cyja4 proxy "GRPC" to URL map psm-interop-url-map-20240111-1936-cyja4
I0111 19:37:38.345857 139848621117440 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1936-cyja4
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1936-cyja4
validate_for_proxyless: true
...
I0111 19:37:38.870654 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001858347-60eb0ae1bb383-7a47c5d4-e838477a
I0111 19:37:41.133113 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.03_create_target_proxy ---
I0111 19:37:41.133332 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0111 19:37:41.133435 139848621117440 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1936-cyja4" in network "default-vpc": 0.0.0.0:41089 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1936-cyja4
I0111 19:37:41.135988 139848621117440 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1936-cyja4
network: global/networks/default-vpc
portRange: 41089
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1936-cyja4
...
I0111 19:37:42.121852 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001861138-60eb0ae4646a8-aac284d3-433fcac3
I0111 19:37:57.333106 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.04_create_forwarding_rule ---
I0111 19:37:57.333347 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0111 19:37:57.333483 139848621117440 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1936-cyja4: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0111 19:37:57.333723 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1936-cyja4%22;timeRange=2024-01-11T19:37:57.333568Z%2F2024-01-11T20:07:57.333568Z?project=grpc-testing
I0111 19:37:57.338714 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1936-cyja4
labels:
name: psm-interop-server-20240111-1936-cyja4
owner: xds-k8s-interop-test
...
I0111 19:37:57.419703 139848621117440 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1936-cyja4 created
I0111 19:37:57.422587 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1936-cyja4-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:37:57.485134 139848621117440 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:37:57.485352 139848621117440 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1936-cyja4
I0111 19:37:57.582336 139848621117440 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1936-cyja4: neg_name=psm-interop-server-20240111-1936-cyja4-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:37:57.582550 139848621117440 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1936-cyja4/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:37:58.342348 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:37:58.401875 139848621117440 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:37:58.410441 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1936-cyja4
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4soyczp8
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1936-cyja4-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:37:58.478842 139848621117440 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:37:58.479103 139848621117440 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0111 19:37:58.634901 139848621117440 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-5cfd6545d6-5hmdg', 'psm-grpc-server-5cfd6545d6-9ckr4', 'psm-grpc-server-5cfd6545d6-bgjgs']
I0111 19:37:58.635306 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-5hmdg to start
I0111 19:38:01.947062 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-5hmdg ready, IP: 10.120.3.99
I0111 19:38:01.947337 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-5hmdg to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-5hmdg.log
I0111 19:38:01.947907 139848528160320 k8s_log_collector.py:70] Starting log collection thread 139848528160320 for psm-grpc-server-5cfd6545d6-5hmdg
I0111 19:38:01.948068 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-9ckr4 to start
I0111 19:38:02.052683 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-9ckr4 ready, IP: 10.120.0.65
I0111 19:38:02.052969 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-9ckr4 to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-9ckr4.log
I0111 19:38:02.053452 139848519767616 k8s_log_collector.py:70] Starting log collection thread 139848519767616 for psm-grpc-server-5cfd6545d6-9ckr4
I0111 19:38:02.053621 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-5cfd6545d6-bgjgs to start
I0111 19:38:02.231600 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-server-5cfd6545d6-bgjgs ready, IP: 10.120.2.167
I0111 19:38:02.231879 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-5cfd6545d6-bgjgs to affinity_test/test_app_logs/psm-interop-server-20240111-1936-cyja4_psm-grpc-server-5cfd6545d6-bgjgs.log
I0111 19:38:02.232368 139848511374912 k8s_log_collector.py:70] Starting log collection thread 139848511374912 for psm-grpc-server-5cfd6545d6-bgjgs
I0111 19:38:02.232539 139848621117440 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0111 19:38:08.634159 139848621117440 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0111 19:38:08.634728 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.05_start_test_servers ---
I0111 19:38:08.634852 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0111 19:38:08.670674 139848621117440 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:38:08.949388 139848621117440 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1936-cyja4-psm-grpc-server" in zone us-central1-a
I0111 19:38:08.949620 139848621117440 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1936-cyja4: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1936-cyja4-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1936-cyja4-psm-grpc-server', zone='us-central1-a')}
I0111 19:38:08.953987 139848621117440 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1936-cyja4-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:38:09.961627 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705001888956-60eb0afeec086-0cabeb4c-4f57f99f
I0111 19:38:57.624413 139848621117440 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1936-cyja4 to report all backends healthy: ['psm-interop-server-20240111-1936-cyja4-psm-grpc-server']
I0111 19:39:10.691787 139848621117440 compute.py:494] Backend psm-interop-server-20240111-1936-cyja4-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:39:14.696536 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.06_add_server_backends_to_backend_services ---
I0111 19:39:14.696756 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0111 19:39:14.696866 139848621117440 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1936-cyja4: server_target=xds:///psm-grpc-server:41089 rpc=EmptyCall qps=25 metadata='EmptyCall:xds_md:123' secure_mode=False print_response=False
I0111 19:39:14.697126 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1936-cyja4%22;timeRange=2024-01-11T19:39:14.696992Z%2F2024-01-11T20:09:14.696992Z?project=grpc-testing
I0111 19:39:14.698979 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1936-cyja4
labels:
name: psm-interop-client-20240111-1936-cyja4
owner: xds-k8s-interop-test
...
I0111 19:39:14.740170 139848621117440 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1936-cyja4 created
I0111 19:39:14.740427 139848621117440 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1936-cyja4/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:39:15.314923 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1936-cyja4
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:39:15.374793 139848621117440 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:39:15.383351 139848621117440 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1936-cyja4
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-yf9833wq
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:41089"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=EmptyCall"
- "--metadata=EmptyCall:xds_md:123"
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:39:15.453018 139848621117440 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:39:15.453255 139848621117440 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:39:15.567415 139848621117440 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6c4558687b-m6qdq']
I0111 19:39:15.567699 139848621117440 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6c4558687b-m6qdq to start
I0111 19:39:18.814490 139848621117440 k8s_base_runner.py:820] Pod psm-grpc-client-6c4558687b-m6qdq ready, IP: 10.120.4.247
I0111 19:39:18.814749 139848621117440 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6c4558687b-m6qdq to affinity_test/test_app_logs/psm-interop-client-20240111-1936-cyja4_psm-grpc-client-6c4558687b-m6qdq.log
I0111 19:39:18.815193 139848295904832 k8s_log_collector.py:70] Starting log collection thread 139848295904832 for psm-grpc-client-6c4558687b-m6qdq
I0111 19:39:18.815328 139848621117440 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:39:20.033506 139848621117440 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:39:20.034057 139848621117440 client_app.py:284] [psm-grpc-client-6c4558687b-m6qdq] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:39:20.137614 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:30.183244 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:40.229298 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:39:50.274333 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:00.319302 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:16.368706 139848621117440 client_app.py:312] [psm-grpc-client-6c4558687b-m6qdq] xDS control plane channel:
I0111 19:40:18.403790 139848621117440 client_app.py:323] [psm-grpc-client-6c4558687b-m6qdq] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:40:18.404110 139848621117440 client_app.py:294] [psm-grpc-client-6c4558687b-m6qdq] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:40:18.404378 139848621117440 client_app.py:244] [psm-grpc-client-6c4558687b-m6qdq] Waiting to report a READY channel to xds:///psm-grpc-server:41089
I0111 19:40:18.436961 139848621117440 client_app.py:371] [psm-grpc-client-6c4558687b-m6qdq] Server channel:
I0111 19:40:18.534163 139848621117440 client_app.py:384] [psm-grpc-client-6c4558687b-m6qdq] Found subchannel in state READY:
I0111 19:40:18.534593 139848621117440 client_app.py:255] [psm-grpc-client-6c4558687b-m6qdq] Channel to xds:///psm-grpc-server:41089 transitioned to state READY:
I0111 19:40:18.534864 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:18.569118 139848621117440 affinity_test.py:108] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/990b5fc1-3a6e-460f-b02e-864bf8a72be6",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:40:18.037648090Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:41089"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:40:18.037824090Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "1705002008279536596",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1936-cyja4"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:40:18.037374940Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "2",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7886369199152299386_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.2.167",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.99",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.65",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:40:12.460373819Z",
"clientStatus": "ACKED"
}
]
}
I0111 19:40:18.569902 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.07_start_test_client ---
I0111 19:40:18.570007 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0111 19:40:18.570116 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:18.603873 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.08_test_client_xds_config_exists ---
I0111 19:40:18.604084 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0111 19:40:18.604303 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:22.628929 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-9ckr4: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-9ckr4: 100
I0111 19:40:22.629190 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.09_test_server_received_rpcs_from_test_client ---
I0111 19:40:22.629281 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0111 19:40:22.629405 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:26.628989 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-9ckr4: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-9ckr4: 100
I0111 19:40:26.759898 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.10_first_100_affinity_rpcs_pick_same_backend ---
I0111 19:40:26.760120 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0111 19:40:26.760218 139848621117440 server_app.py:98] [psm-grpc-server-5cfd6545d6-9ckr4] >> Setting health status to NOT_SERVING
I0111 19:40:26.760625 139848621117440 grpc.py:75] [psm-grpc-server-5cfd6545d6-9ckr4:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.858044 139848621117440 grpc.py:75] [psm-grpc-server-5cfd6545d6-9ckr4:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.890568 139848621117440 server_app.py:102] [psm-grpc-server-5cfd6545d6-9ckr4] << Health status status: NOT_SERVING
I0111 19:40:26.890804 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.11_turn_down_server_in_use ---
I0111 19:40:26.890896 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0111 19:40:26.891036 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:26.924497 139848621117440 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0111 19:40:36.934947 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:36.970628 139848621117440 affinity_test.py:172] CSDS got unexpected endpoints, will retry after 10 seconds
I0111 19:40:46.981101 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 19:40:47.016489 139848621117440 affinity_test.py:185] Client received CSDS response: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/990b5fc1-3a6e-460f-b02e-864bf8a72be6",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.247",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:41089",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:40:37.901593167Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:41089"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
},
"hashPolicy": [
{
"header": {
"headerName": "xds_md"
}
}
]
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1936-cyja4_0_psm-grpc-server:41089-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:40:37.902302107Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "1705002031557437118",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"lbPolicy": "RING_HASH",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1936-cyja4"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1936-cyja4",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:40:37.901300707Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"versionInfo": "6",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_690611996802809257",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_7886369199152299386_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.2.167",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.3.99",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
},
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.65",
"portValue": 8080
}
}
},
"healthStatus": "UNHEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:40:42.460178970Z",
"clientStatus": "ACKED"
}
]
}
I0111 19:40:47.017225 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_wait_for_unhealth_status_propagation ---
I0111 19:40:47.017334 139848621117440 xds_k8s_testcase.py:269] --- Starting subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0111 19:40:47.017457 139848621117440 grpc.py:75] [psm-grpc-client-6c4558687b-m6qdq:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:40:51.028639 139848621117440 xds_k8s_testcase.py:601] [psm-grpc-client-6c4558687b-m6qdq] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-5cfd6545d6-bgjgs: 100
rpcs_by_method:
EmptyCall:
psm-grpc-server-5cfd6545d6-bgjgs: 100
I0111 19:40:51.028964 139848621117440 xds_k8s_testcase.py:274] --- Finished subTest __main__.AffinityTest.test_affinity.12_next_100_affinity_rpcs_pick_different_backend ---
I0111 19:40:51.029119 139848621117440 xds_k8s_testcase.py:687] ----- TestMethod __main__.AffinityTest.test_affinity teardown -----
I0111 19:40:51.280316 139848621117440 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1936-cyja4"
I0111 19:40:51.998042 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002051284-60eb0b99badda-38aac552-bf4f7a65
I0111 19:41:15.804803 139848621117440 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1936-cyja4"
I0111 19:41:15.928511 139848621117440 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1936-cyja4"
I0111 19:41:16.476036 139848621117440 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1936-cyja4"
I0111 19:41:16.920675 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002076480-60eb0bb1c2488-43294e1a-18c74b81
I0111 19:41:19.139220 139848621117440 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1936-cyja4"
I0111 19:41:19.646523 139848621117440 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1936-cyja4"
I0111 19:41:20.403494 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002079652-60eb0bb4c8941-c33aae89-3ef450ca
I0111 19:41:24.818637 139848621117440 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1936-cyja4"
I0111 19:41:25.276229 139848621117440 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1936-cyja4"
I0111 19:41:25.903390 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002085282-60eb0bba270f6-a55f8819-c1cfa9ba
I0111 19:41:58.339715 139848621117440 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1936-cyja4"
I0111 19:41:58.822424 139848621117440 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1936-cyja4"
I0111 19:41:59.310904 139848621117440 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1936-cyja4"
I0111 19:41:59.927478 139848621117440 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002119406-60eb0bdab24a6-c8441f46-1b4e11f9
I0111 19:42:04.437045 139848621117440 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 19:42:14.630590 139848621117440 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 19:42:14.630914 139848621117440 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1936-cyja4/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:42:15.189674 139848621117440 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 19:42:15.305894 139848621117440 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 19:42:15.306201 139848621117440 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1936-cyja4
I0111 19:42:45.491417 139848621117440 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1936-cyja4 deleted
I0111 19:42:45.491656 139848621117440 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 19:42:55.722656 139848621117440 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 19:42:55.722935 139848621117440 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 19:42:55.837761 139848621117440 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 19:42:55.838038 139848621117440 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1936-cyja4/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:42:57.888352 139848621117440 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 19:42:58.003778 139848621117440 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 19:42:58.004035 139848621117440 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1936-cyja4
I0111 19:44:28.326913 139848621117440 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1936-cyja4 deleted
I0111 19:44:28.327356 139848621117440 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 19:44:28.327643 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-yf9833wq:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1936-cyja4%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-yf9833wq%22;timeRange=2024-01-11T19:39:14.696984Z%2F2024-01-11T19:42:45.491625Z?project=grpc-testing
I0111 19:44:28.327806 139848621117440 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-4soyczp8:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1936-cyja4%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-4soyczp8%22;timeRange=2024-01-11T19:37:57.333559Z%2F2024-01-11T19:44:28.327190Z?project=grpc-testing
[ OK ] AffinityTest.test_affinity
I0111 19:44:28.328090 139848621117440 base_testcase.py:52] ----- TestCase __main__.AffinityTest.test_affinity PASSED -----
----------------------------------------------------------------------
Ran 1 test in 474.130s
OK
+ for test in "${test_suites[@]}"
+ run_test api_listener_test
+ local test_name=api_listener_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test'
+ set -x
+ python -m tests.api_listener_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:44:29.736641 140453231611904 xds_k8s_testcase.py:157] ----- Testing ApiListenerTest -----
I0111 19:44:29.736788 140453231611904 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:44:29.737011 140453231611904 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:44:29.754577 140453231611904 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:44:29.771473 140453231611904 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0111 19:44:29.776247 140453231611904 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1944-hkz7f
I0111 19:44:30.091493 140453231611904 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/api_listener_test
I0111 19:44:30.534574 140453231611904 xds_k8s_testcase.py:663] Found unused xds port: 58545
I0111 19:44:30.534909 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0111 19:44:30.534989 140453231611904 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1944-hkz7f"
I0111 19:44:30.538130 140453231611904 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1944-hkz7f
type: GRPC
...
I0111 19:44:31.270478 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002270610-60eb0c6ae555d-094ac120-66e891ea
I0111 19:44:33.645896 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0111 19:44:33.646111 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0111 19:44:33.646182 140453231611904 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1944-hkz7f"
I0111 19:44:33.654427 140453231611904 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1944-hkz7f
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-1944-hkz7f
protocol: GRPC
...
I0111 19:44:34.509898 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002273656-60eb0c6dccd5b-d016d964-7fbdcb46
I0111 19:44:54.166541 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0111 19:44:54.166707 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0111 19:44:54.166772 140453231611904 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1944-hkz7f": psm-grpc-server:58545 -> psm-interop-backend-service-20240111-1944-hkz7f
I0111 19:44:54.178301 140453231611904 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
hostRules:
- hosts:
- psm-grpc-server:58545
pathMatcher: psm-interop-path-matcher-20240111-1944-hkz7f
name: psm-interop-url-map-20240111-1944-hkz7f
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
name: psm-interop-path-matcher-20240111-1944-hkz7f
...
I0111 19:44:54.784215 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002294180-60eb0c815fb18-12ef2ed4-31d69a12
I0111 19:44:57.060353 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0111 19:44:57.060564 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0111 19:44:57.060652 140453231611904 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1944-hkz7f proxy "GRPC" to URL map psm-interop-url-map-20240111-1944-hkz7f
I0111 19:44:57.062730 140453231611904 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1944-hkz7f
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1944-hkz7f
validate_for_proxyless: true
...
I0111 19:44:57.639843 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002297064-60eb0c841fc4a-45ffd04c-71c75153
I0111 19:44:59.951745 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0111 19:44:59.951989 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0111 19:44:59.952111 140453231611904 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f" in network "default-vpc": 0.0.0.0:58545 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1944-hkz7f
I0111 19:44:59.954758 140453231611904 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1944-hkz7f
network: global/networks/default-vpc
portRange: 58545
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1944-hkz7f
...
I0111 19:45:00.778454 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002299956-60eb0c86e1d81-30a568ab-d89e7374
I0111 19:45:45.941831 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0111 19:45:45.942088 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0111 19:45:45.942193 140453231611904 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1944-hkz7f: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:45:45.942449 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1944-hkz7f%22;timeRange=2024-01-11T19:45:45.942290Z%2F2024-01-11T20:15:45.942290Z?project=grpc-testing
I0111 19:45:45.947467 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1944-hkz7f
labels:
name: psm-interop-server-20240111-1944-hkz7f
owner: xds-k8s-interop-test
...
I0111 19:45:46.095828 140453231611904 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1944-hkz7f created
I0111 19:45:46.098723 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1944-hkz7f-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:45:46.169998 140453231611904 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:45:46.170229 140453231611904 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1944-hkz7f
I0111 19:45:46.263880 140453231611904 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1944-hkz7f: neg_name=psm-interop-server-20240111-1944-hkz7f-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:45:46.264085 140453231611904 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1944-hkz7f/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:45:46.995814 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:45:47.064201 140453231611904 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:45:47.073744 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1944-hkz7f
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-5emyk8d4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1944-hkz7f-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:45:47.142991 140453231611904 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:45:47.143227 140453231611904 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 19:45:47.246189 140453231611904 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6d889f745f-w8d94']
I0111 19:45:47.246444 140453231611904 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6d889f745f-w8d94 to start
I0111 19:45:50.484474 140453231611904 k8s_base_runner.py:820] Pod psm-grpc-server-6d889f745f-w8d94 ready, IP: 10.120.0.66
I0111 19:45:50.484750 140453231611904 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6d889f745f-w8d94 to api_listener_test/test_app_logs/psm-interop-server-20240111-1944-hkz7f_psm-grpc-server-6d889f745f-w8d94.log
I0111 19:45:50.485298 140453138130496 k8s_log_collector.py:70] Starting log collection thread 140453138130496 for psm-grpc-server-6d889f745f-w8d94
I0111 19:45:50.485541 140453231611904 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 19:45:56.837117 140453231611904 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 19:45:56.837530 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0111 19:45:56.837635 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0111 19:45:56.873633 140453231611904 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:45:57.189773 140453231611904 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1944-hkz7f-psm-grpc-server" in zone us-central1-a
I0111 19:45:57.190027 140453231611904 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1944-hkz7f: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1944-hkz7f-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1944-hkz7f-psm-grpc-server', zone='us-central1-a')}
I0111 19:45:57.194467 140453231611904 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1944-hkz7f-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:45:58.067478 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002357197-60eb0cbd78beb-d2177fa3-3bc3c373
I0111 19:46:54.211319 140453231611904 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1944-hkz7f to report all backends healthy: ['psm-interop-server-20240111-1944-hkz7f-psm-grpc-server']
I0111 19:47:07.208019 140453231611904 compute.py:494] Backend psm-interop-server-20240111-1944-hkz7f-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:47:11.212684 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0111 19:47:11.212927 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0111 19:47:11.213034 140453231611904 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1944-hkz7f: server_target=xds:///psm-grpc-server:58545 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 19:47:11.213273 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1944-hkz7f%22;timeRange=2024-01-11T19:47:11.213132Z%2F2024-01-11T20:17:11.213132Z?project=grpc-testing
I0111 19:47:11.215191 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1944-hkz7f
labels:
name: psm-interop-client-20240111-1944-hkz7f
owner: xds-k8s-interop-test
...
I0111 19:47:11.257156 140453231611904 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1944-hkz7f created
I0111 19:47:11.257391 140453231611904 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1944-hkz7f/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:47:11.863847 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1944-hkz7f
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:47:11.930180 140453231611904 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:47:12.003937 140453231611904 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1944-hkz7f
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-afmvfn87
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:58545"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:47:12.076130 140453231611904 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:47:12.076387 140453231611904 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:47:12.180996 140453231611904 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-fcdb59d46-8vr7t']
I0111 19:47:12.181296 140453231611904 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-fcdb59d46-8vr7t to start
I0111 19:47:15.418415 140453231611904 k8s_base_runner.py:820] Pod psm-grpc-client-fcdb59d46-8vr7t ready, IP: 10.120.4.248
I0111 19:47:15.418705 140453231611904 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-fcdb59d46-8vr7t to api_listener_test/test_app_logs/psm-interop-client-20240111-1944-hkz7f_psm-grpc-client-fcdb59d46-8vr7t.log
I0111 19:47:15.419272 140453129737792 k8s_log_collector.py:70] Starting log collection thread 140453129737792 for psm-grpc-client-fcdb59d46-8vr7t
I0111 19:47:15.419488 140453231611904 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:47:16.560678 140453231611904 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:47:16.561240 140453231611904 client_app.py:284] [psm-grpc-client-fcdb59d46-8vr7t] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:47:16.669472 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:26.716017 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:36.761762 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:46.807155 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:47:56.852962 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:12.902960 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:37.963985 140453231611904 client_app.py:312] [psm-grpc-client-fcdb59d46-8vr7t] xDS control plane channel:
I0111 19:48:39.999708 140453231611904 client_app.py:323] [psm-grpc-client-fcdb59d46-8vr7t] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:48:39.999959 140453231611904 client_app.py:294] [psm-grpc-client-fcdb59d46-8vr7t] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:48:40.000201 140453231611904 client_app.py:244] [psm-grpc-client-fcdb59d46-8vr7t] Waiting to report a READY channel to xds:///psm-grpc-server:58545
I0111 19:48:40.033413 140453231611904 client_app.py:371] [psm-grpc-client-fcdb59d46-8vr7t] Server channel:
I0111 19:48:40.066776 140453231611904 client_app.py:384] [psm-grpc-client-fcdb59d46-8vr7t] Found subchannel in state READY:
I0111 19:48:40.067118 140453231611904 client_app.py:255] [psm-grpc-client-fcdb59d46-8vr7t] Channel to xds:///psm-grpc-server:58545 transitioned to state READY:
I0111 19:48:40.067219 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0111 19:48:40.067281 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0111 19:48:40.067470 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:48:40.103037 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.08_test_client_xds_config_exists ---
I0111 19:48:40.103269 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0111 19:48:40.103491 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:48:44.106116 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:48:44.106467 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.09_test_server_received_rpcs ---
I0111 19:48:44.106555 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0111 19:48:44.106630 140453231611904 traffic_director.py:540] Creating alternative URL map "psm-interop-url-map-alt-20240111-1944-hkz7f": psm-grpc-server:58545 -> psm-interop-backend-service-20240111-1944-hkz7f
I0111 19:48:44.111719 140453231611904 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
hostRules:
- hosts:
- psm-grpc-server:58545
pathMatcher: psm-interop-path-matcher-20240111-1944-hkz7f
name: psm-interop-url-map-alt-20240111-1944-hkz7f
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f
name: psm-interop-path-matcher-20240111-1944-hkz7f
...
I0111 19:48:44.827340 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002524115-60eb0d5ca80cf-223e9215-6b043585
I0111 19:48:47.093788 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.10_create_alternate_url_map ---
I0111 19:48:47.094034 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0111 19:48:47.094131 140453231611904 traffic_director.py:613] Creating alternative target GRPC proxy "psm-interop-target-proxy-alt-20240111-1944-hkz7f" to URL map psm-interop-url-map-alt-20240111-1944-hkz7f
I0111 19:48:47.096434 140453231611904 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-alt-20240111-1944-hkz7f
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-alt-20240111-1944-hkz7f
validate_for_proxyless: false
...
I0111 19:48:47.748824 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002527098-60eb0d5f805fd-db12f20a-ff89db66
I0111 19:48:50.037725 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.11_create_alternate_target_proxy ---
I0111 19:48:50.037978 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0111 19:48:50.038092 140453231611904 traffic_director.py:683] Creating alternative forwarding rule "psm-interop-forwarding-rule-alt-20240111-1944-hkz7f" in network "default-vpc": 10.10.10.10:58545 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240111-1944-hkz7f
I0111 19:48:50.040755 140453231611904 compute.py:537] Creating compute resource:
---
IPAddress: 10.10.10.10
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-alt-20240111-1944-hkz7f
network: global/networks/default-vpc
portRange: 58545
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-alt-20240111-1944-hkz7f
...
I0111 19:48:50.916442 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002530043-60eb0d624f53b-ddea5fe5-7646d95d
I0111 19:49:29.771052 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.12_create_alternate_forwarding_rule ---
I0111 19:49:29.771300 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0111 19:49:29.771491 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:49:33.785936 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:49:33.786256 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:49:33.820180 140453231611904 api_listener_test.py:110] received client config from CSDS with two url maps, dump config: {
"node": {
"id": "projects/830293263384/networks/default-vpc/nodes/2381cab1-578e-4fc1-89a4-1f961af362ab",
"cluster": "cluster",
"metadata": {
"INSTANCE_IP": "10.120.4.248",
"TRAFFICDIRECTOR_NETWORK_NAME": "default-vpc",
"TRAFFICDIRECTOR_GCP_PROJECT_NUMBER": "830293263384"
},
"locality": {
"zone": "us-central1-a"
},
"userAgentName": "gRPC Go",
"userAgentVersion": "1.61.0-dev",
"clientFeatures": [
"envoy.lb.does_not_support_overprovisioning",
"xds.config.resource-in-sotw"
]
},
"genericXdsConfigs": [
{
"typeUrl": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"versionInfo": "8",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment",
"clusterName": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"endpoints": [
{
"locality": {
"subZone": "ib:us-central1-a_6346379817286237061_neg"
},
"lbEndpoints": [
{
"endpoint": {
"address": {
"socketAddress": {
"address": "10.120.0.66",
"portValue": 8080
}
}
},
"healthStatus": "HEALTHY"
}
],
"loadBalancingWeight": 1000000
}
]
},
"lastUpdated": "2024-01-11T19:49:31.317665801Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:58545",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
"name": "psm-grpc-server:58545",
"apiListener": {
"apiListener": {
"@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
"statPrefix": "trafficdirector",
"rds": {
"configSource": {
"ads": {},
"resourceApiVersion": "V3"
},
"routeConfigName": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545"
},
"httpFilters": [
{
"name": "envoy.filters.http.fault",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
}
},
{
"name": "envoy.filters.http.router",
"typedConfig": {
"@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router",
"suppressEnvoyHeaders": true
}
}
],
"normalizePath": true,
"mergeSlashes": true
}
}
},
"lastUpdated": "2024-01-11T19:49:30.867656445Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.route.v3.RouteConfiguration",
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545",
"virtualHosts": [
{
"domains": [
"psm-grpc-server:58545"
],
"routes": [
{
"match": {
"prefix": ""
},
"route": {
"cluster": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"timeout": "30s",
"retryPolicy": {
"retryOn": "gateway-error",
"numRetries": 1,
"perTryTimeout": "30s"
}
},
"name": "URL_MAP/830293263384_psm-interop-url-map-20240111-1944-hkz7f_0_psm-grpc-server:58545-route-0"
}
]
}
]
},
"lastUpdated": "2024-01-11T19:49:30.867983355Z",
"clientStatus": "ACKED"
},
{
"typeUrl": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"versionInfo": "1705002563355708307",
"xdsConfig": {
"@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
"name": "cloud-internal-istio:cloud_mp_830293263384_5807702394796899277",
"type": "EDS",
"edsClusterConfig": {
"edsConfig": {
"ads": {},
"initialFetchTimeout": "15s",
"resourceApiVersion": "V3"
}
},
"connectTimeout": "30s",
"circuitBreakers": {
"thresholds": [
{
"maxConnections": 2147483647,
"maxPendingRequests": 2147483647,
"maxRequests": 2147483647,
"maxRetries": 2147483647
}
]
},
"http2ProtocolOptions": {
"maxConcurrentStreams": 100
},
"metadata": {
"filterMetadata": {
"com.google.trafficdirector": {
"backend_service_project_number": 830293263384.0,
"backend_service_name": "psm-interop-backend-service-20240111-1944-hkz7f"
}
}
},
"commonLbConfig": {
"healthyPanicThreshold": {
"value": 1.0
},
"localityWeightedLbConfig": {}
},
"altStatName": "/projects/830293263384/global/backendServices/psm-interop-backend-service-20240111-1944-hkz7f",
"lrsServer": {
"self": {}
}
},
"lastUpdated": "2024-01-11T19:49:30.867038635Z",
"clientStatus": "ACKED"
}
]
}, rds version: 1705002563355708307
I0111 19:49:33.820889 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.13_test_server_received_rpcs_with_two_url_maps ---
I0111 19:49:33.821023 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0111 19:49:33.821107 140453231611904 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f"
I0111 19:49:34.477007 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002573824-60eb0d8c101d3-df182fdf-f9bca504
I0111 19:49:47.543639 140453231611904 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:49:47.987516 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002587546-60eb0d9926473-a3f9b8c2-5b3c5804
I0111 19:49:50.305828 140453231611904 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1944-hkz7f"
I0111 19:49:51.012564 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002590311-60eb0d9bc93a6-f50a21a0-630c58d6
I0111 19:49:53.262507 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.14_delete_one_url_map_target_proxy_forwarding_rule ---
I0111 19:49:53.262724 140453231611904 xds_k8s_testcase.py:269] --- Starting subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0111 19:49:53.263120 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:49:57.266071 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:49:57.266360 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:49:57.300235 140453231611904 xds_k8s_testcase.py:552] Routing config not propagated yet. Retrying.
I0111 19:49:57.300778 140453231611904 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0111 19:49:59.303178 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:03.306487 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:03.306796 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:50:03.340737 140453231611904 xds_k8s_testcase.py:552] Routing config not propagated yet. Retrying.
I0111 19:50:03.341084 140453231611904 retryers.py:206] Retrying None in 2.0 seconds as it raised TdPropagationRetryableError: CSDS not get updated routing config corresponding to the second set of url maps.
I0111 19:50:05.343477 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:09.346052 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:09.346329 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:50:09.380117 140453231611904 grpc.py:75] [psm-grpc-client-fcdb59d46-8vr7t:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), wait_for_ready=True, timeout=1200)
I0111 19:50:13.386589 140453231611904 xds_k8s_testcase.py:601] [psm-grpc-client-fcdb59d46-8vr7t] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6d889f745f-w8d94: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6d889f745f-w8d94: 100
I0111 19:50:13.386839 140453231611904 xds_k8s_testcase.py:561] [SUCCESS] Confirmed successful RPC with the updated routing config, version=1705002585732599240
I0111 19:50:13.387053 140453231611904 xds_k8s_testcase.py:274] --- Finished subTest __main__.ApiListenerTest.test_api_listener.15_test_server_continues_to_receive_rpcs ---
I0111 19:50:13.387200 140453231611904 xds_k8s_testcase.py:687] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0111 19:50:13.610906 140453231611904 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1944-hkz7f"
I0111 19:50:13.740448 140453231611904 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1944-hkz7f"
I0111 19:50:14.356550 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002613743-60eb0db221fba-56ec9e65-c03feb12
I0111 19:50:27.317075 140453231611904 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:50:27.835503 140453231611904 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1944-hkz7f"
I0111 19:50:28.257371 140453231611904 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1944-hkz7f"
I0111 19:50:28.658040 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002628260-60eb0dbffa341-681f5ec5-8358f919
I0111 19:50:30.900532 140453231611904 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1944-hkz7f"
I0111 19:50:31.314488 140453231611904 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1944-hkz7f"
I0111 19:50:32.053630 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002631320-60eb0dc2e546d-89822ca0-37ace6c1
I0111 19:50:34.395989 140453231611904 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1944-hkz7f"
I0111 19:50:35.074580 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002634401-60eb0dc5d574d-ac61f050-147c35b2
I0111 19:50:58.845590 140453231611904 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1944-hkz7f"
I0111 19:50:59.289386 140453231611904 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1944-hkz7f"
I0111 19:50:59.701323 140453231611904 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1944-hkz7f"
I0111 19:51:00.204261 140453231611904 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002659779-60eb0dde0930e-859f9209-3c0940dd
I0111 19:51:04.714053 140453231611904 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 19:51:14.894882 140453231611904 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 19:51:14.895221 140453231611904 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1944-hkz7f/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:51:15.497268 140453231611904 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 19:51:15.619240 140453231611904 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 19:51:15.619474 140453231611904 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1944-hkz7f
I0111 19:51:45.805962 140453231611904 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1944-hkz7f deleted
I0111 19:51:45.806253 140453231611904 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 19:51:55.980636 140453231611904 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 19:51:55.980950 140453231611904 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 19:51:56.097112 140453231611904 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 19:51:56.097378 140453231611904 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1944-hkz7f/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:51:56.825804 140453231611904 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 19:51:56.940458 140453231611904 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 19:51:56.940704 140453231611904 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1944-hkz7f
I0111 19:53:27.296045 140453231611904 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1944-hkz7f deleted
I0111 19:53:27.296488 140453231611904 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 19:53:27.296767 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-afmvfn87:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1944-hkz7f%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-afmvfn87%22;timeRange=2024-01-11T19:47:11.213124Z%2F2024-01-11T19:51:45.806221Z?project=grpc-testing
I0111 19:53:27.296961 140453231611904 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-5emyk8d4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1944-hkz7f%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-5emyk8d4%22;timeRange=2024-01-11T19:45:45.942281Z%2F2024-01-11T19:53:27.296309Z?project=grpc-testing
[ OK ] ApiListenerTest.test_api_listener
I0111 19:53:27.297256 140453231611904 base_testcase.py:52] ----- TestCase __main__.ApiListenerTest.test_api_listener PASSED -----
----------------------------------------------------------------------
Ran 1 test in 537.564s
OK
+ for test in "${test_suites[@]}"
+ run_test change_backend_service_test
+ local test_name=change_backend_service_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test'
+ set -x
+ python -m tests.change_backend_service_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 19:53:28.657442 139660000452608 xds_k8s_testcase.py:157] ----- Testing ChangeBackendServiceTest -----
I0111 19:53:28.657607 139660000452608 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 19:53:28.657845 139660000452608 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 19:53:28.674901 139660000452608 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 19:53:28.691626 139660000452608 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0111 19:53:28.696330 139660000452608 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-1953-20o1e
I0111 19:53:28.991961 139660000452608 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/change_backend_service_test
I0111 19:53:29.436016 139660000452608 xds_k8s_testcase.py:663] Found unused xds port: 49735
I0111 19:53:29.438322 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0111 19:53:29.438541 139660000452608 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-1953-20o1e"
I0111 19:53:29.441645 139660000452608 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-1953-20o1e
type: GRPC
...
I0111 19:53:30.186767 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002809520-60eb0e6cd72c4-e396e11c-56554943
I0111 19:53:32.504934 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0111 19:53:32.505191 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0111 19:53:32.505293 139660000452608 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-1953-20o1e"
I0111 19:53:32.513587 139660000452608 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1953-20o1e
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-1953-20o1e
protocol: GRPC
...
I0111 19:53:33.239455 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002812515-60eb0e6fb2507-0338db9a-500bc107
I0111 19:54:16.438918 139660000452608 traffic_director.py:309] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20240111-1953-20o1e"
I0111 19:54:16.444117 139660000452608 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-1953-20o1e
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20240111-1953-20o1e
protocol: GRPC
...
I0111 19:54:17.231704 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002856446-60eb0e9997893-419d1125-1d9cdd97
I0111 19:54:38.813266 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0111 19:54:38.813483 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0111 19:54:38.813588 139660000452608 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-1953-20o1e": psm-grpc-server:49735 -> psm-interop-backend-service-20240111-1953-20o1e
I0111 19:54:38.825136 139660000452608 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1953-20o1e
hostRules:
- hosts:
- psm-grpc-server:49735
pathMatcher: psm-interop-path-matcher-20240111-1953-20o1e
name: psm-interop-url-map-20240111-1953-20o1e
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-1953-20o1e
name: psm-interop-path-matcher-20240111-1953-20o1e
...
I0111 19:54:39.573349 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002878827-60eb0eaeefa6c-a7d7ca92-dc7ea65d
I0111 19:54:41.939240 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0111 19:54:41.939477 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0111 19:54:41.939605 139660000452608 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-1953-20o1e proxy "GRPC" to URL map psm-interop-url-map-20240111-1953-20o1e
I0111 19:54:41.941506 139660000452608 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-1953-20o1e
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-1953-20o1e
validate_for_proxyless: true
...
I0111 19:54:42.708473 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002881943-60eb0eb1e8929-0bc83180-3af59637
I0111 19:54:44.915428 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0111 19:54:44.915642 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0111 19:54:44.915730 139660000452608 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-1953-20o1e" in network "default-vpc": 0.0.0.0:49735 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1953-20o1e
I0111 19:54:44.918292 139660000452608 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-1953-20o1e
network: global/networks/default-vpc
portRange: 49735
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-1953-20o1e
...
I0111 19:54:45.757917 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002884920-60eb0eb4bf3e7-238ac4a3-f7f3b74a
I0111 19:54:58.990261 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0111 19:54:58.990529 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0111 19:54:58.990638 139660000452608 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-1953-20o1e: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:54:58.990908 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22;timeRange=2024-01-11T19:54:58.990740Z%2F2024-01-11T20:24:58.990740Z?project=grpc-testing
I0111 19:54:58.996098 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-1953-20o1e
labels:
name: psm-interop-server-20240111-1953-20o1e
owner: xds-k8s-interop-test
...
I0111 19:54:59.145261 139660000452608 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-1953-20o1e created
I0111 19:54:59.148380 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1953-20o1e-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:54:59.216807 139660000452608 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 19:54:59.217024 139660000452608 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-1953-20o1e
I0111 19:54:59.311780 139660000452608 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-1953-20o1e: neg_name=psm-interop-server-20240111-1953-20o1e-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 19:54:59.311990 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:55:00.119301 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:55:00.179517 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 19:55:00.189339 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-1953-20o1e
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-snef9hve
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1953-20o1e-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:55:00.258826 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 19:55:00.259093 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 19:55:00.366019 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-69cdbd8b49-75d44']
I0111 19:55:00.366358 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-69cdbd8b49-75d44 to start
I0111 19:55:03.606573 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-server-69cdbd8b49-75d44 ready, IP: 10.120.2.171
I0111 19:55:03.606832 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-69cdbd8b49-75d44 to change_backend_service_test/test_app_logs/psm-interop-server-20240111-1953-20o1e_psm-grpc-server-69cdbd8b49-75d44.log
I0111 19:55:03.607274 139659907163712 k8s_log_collector.py:70] Starting log collection thread 139659907163712 for psm-grpc-server-69cdbd8b49-75d44
I0111 19:55:03.607409 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 19:55:08.905764 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 19:55:08.906187 139660000452608 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-1953-20o1e: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 19:55:08.906420 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22;timeRange=2024-01-11T19:55:08.906299Z%2F2024-01-11T20:25:08.906299Z?project=grpc-testing
I0111 19:55:08.906520 139660000452608 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240111-1953-20o1e
I0111 19:55:08.962622 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 19:55:09.009193 139660000452608 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 19:55:09.009463 139660000452608 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-1953-20o1e
I0111 19:55:09.104058 139660000452608 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-1953-20o1e: neg_name=psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0111 19:55:09.104266 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:55:09.721852 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:55:09.762687 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 19:55:09.771005 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-1953-20o1e
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-kp9d1zb5
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:55:09.822514 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 19:55:09.822765 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 19:55:09.968732 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:55:09.969035 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-7468b95b99-5rrqx to start
I0111 19:55:12.164124 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-server-alt-7468b95b99-5rrqx ready, IP: 10.120.7.128
I0111 19:55:12.164401 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-7468b95b99-5rrqx to change_backend_service_test/test_app_logs/psm-interop-server-20240111-1953-20o1e_psm-grpc-server-alt-7468b95b99-5rrqx.log
I0111 19:55:12.164910 139659898771008 k8s_log_collector.py:70] Starting log collection thread 139659898771008 for psm-grpc-server-alt-7468b95b99-5rrqx
I0111 19:55:12.165111 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 19:55:19.629064 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 19:55:19.629516 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0111 19:55:19.629632 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0111 19:55:19.666485 139660000452608 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:55:19.789431 139660000452608 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-1953-20o1e-psm-grpc-server" in zone us-central1-a
I0111 19:55:19.789646 139660000452608 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-1953-20o1e: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1953-20o1e-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server', zone='us-central1-a')}
I0111 19:55:19.794117 139660000452608 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 19:55:20.596027 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002919796-60eb0ed601e69-45ada39b-b9b58665
I0111 19:56:01.838949 139660000452608 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-1953-20o1e to report all backends healthy: ['psm-interop-server-20240111-1953-20o1e-psm-grpc-server']
I0111 19:56:06.356426 139660000452608 compute.py:494] Backend psm-interop-server-20240111-1953-20o1e-psm-grpc-server in zone us-central1-a reported healthy
I0111 19:56:10.399091 139660000452608 traffic_director.py:337] Waiting for Network Endpoint Groups to load endpoints.
I0111 19:56:10.743415 139660000452608 traffic_director.py:340] Loaded NEG "psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt" in zone us-central1-a
I0111 19:56:10.743677 139660000452608 traffic_director.py:347] Adding backends to Backend Service psm-interop-backend-service-alt-20240111-1953-20o1e: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt', zone='us-central1-a')}
I0111 19:56:10.748290 139660000452608 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-1953-20o1e-psm-grpc-server-alt
maxRatePerEndpoint: 5
...
I0111 19:56:11.687826 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705002970750-60eb0f0699daa-742fae08-0a735aed
I0111 19:56:50.470765 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0111 19:56:50.470988 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0111 19:56:50.471108 139660000452608 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-1953-20o1e: server_target=xds:///psm-grpc-server:49735 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 19:56:50.471346 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1953-20o1e%22;timeRange=2024-01-11T19:56:50.471188Z%2F2024-01-11T20:26:50.471188Z?project=grpc-testing
I0111 19:56:50.473242 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-1953-20o1e
labels:
name: psm-interop-client-20240111-1953-20o1e
owner: xds-k8s-interop-test
...
I0111 19:56:50.515286 139660000452608 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-1953-20o1e created
I0111 19:56:50.515504 139660000452608 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1953-20o1e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 19:56:51.078962 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1953-20o1e
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 19:56:51.138321 139660000452608 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 19:56:51.146969 139660000452608 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-1953-20o1e
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6gyraz8u
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:49735"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 19:56:51.218494 139660000452608 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 19:56:51.218724 139660000452608 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 19:56:51.323243 139660000452608 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-bc5fc5bc6-mjjv6']
I0111 19:56:51.323502 139660000452608 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-bc5fc5bc6-mjjv6 to start
I0111 19:56:55.618076 139660000452608 k8s_base_runner.py:820] Pod psm-grpc-client-bc5fc5bc6-mjjv6 ready, IP: 10.120.0.69
I0111 19:56:55.618357 139660000452608 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-bc5fc5bc6-mjjv6 to change_backend_service_test/test_app_logs/psm-interop-client-20240111-1953-20o1e_psm-grpc-client-bc5fc5bc6-mjjv6.log
I0111 19:56:55.618835 139659890378304 k8s_log_collector.py:70] Starting log collection thread 139659890378304 for psm-grpc-client-bc5fc5bc6-mjjv6
I0111 19:56:55.619029 139660000452608 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 19:56:58.841220 139660000452608 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 19:56:58.841778 139660000452608 client_app.py:284] [psm-grpc-client-bc5fc5bc6-mjjv6] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 19:56:58.945901 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:08.991215 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:19.036087 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:29.080876 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:39.125536 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:57:55.174434 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:58:20.233860 139660000452608 client_app.py:312] [psm-grpc-client-bc5fc5bc6-mjjv6] xDS control plane channel:
I0111 19:58:22.268343 139660000452608 client_app.py:323] [psm-grpc-client-bc5fc5bc6-mjjv6] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 19:58:22.268630 139660000452608 client_app.py:294] [psm-grpc-client-bc5fc5bc6-mjjv6] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 19:58:22.268894 139660000452608 client_app.py:244] [psm-grpc-client-bc5fc5bc6-mjjv6] Waiting to report a READY channel to xds:///psm-grpc-server:49735
I0111 19:58:22.300961 139660000452608 client_app.py:371] [psm-grpc-client-bc5fc5bc6-mjjv6] Server channel:
I0111 19:58:22.333167 139660000452608 client_app.py:384] [psm-grpc-client-bc5fc5bc6-mjjv6] Found subchannel in state READY:
I0111 19:58:22.333569 139660000452608 client_app.py:255] [psm-grpc-client-bc5fc5bc6-mjjv6] Channel to xds:///psm-grpc-server:49735 transitioned to state READY:
I0111 19:58:22.333703 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0111 19:58:22.333796 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0111 19:58:22.333996 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 19:58:22.368292 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.08_test_client_xds_config_exists ---
I0111 19:58:22.368509 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0111 19:58:22.368712 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:26.382597 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:26.382882 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.09_test_server_received_rpcs_from_test_client ---
I0111 19:58:26.382986 139660000452608 xds_k8s_testcase.py:269] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0111 19:58:26.383072 139660000452608 traffic_director.py:499] Patching URL map "psm-interop-url-map-20240111-1953-20o1e": psm-grpc-server:49735 -> psm-interop-backend-service-alt-20240111-1953-20o1e
I0111 19:58:26.388541 139660000452608 compute.py:544] Patching compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240111-1953-20o1e
hostRules:
- hosts:
- psm-grpc-server:49735
pathMatcher: psm-interop-path-matcher-20240111-1953-20o1e
name: psm-interop-url-map-20240111-1953-20o1e
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-alt-20240111-1953-20o1e
name: psm-interop-path-matcher-20240111-1953-20o1e
...
I0111 19:58:27.381674 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003106391-60eb0f87f5552-fadee9af-ea78c0f9
I0111 19:58:38.370954 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:38.371234 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:42.383072 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:42.383684 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7468b95b99-5rrqx' not found in {'psm-grpc-server-69cdbd8b49-75d44': 100} : Server psm-grpc-server-alt-7468b95b99-5rrqx did not receive RPCs.
I0111 19:58:43.385011 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:43.385346 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:47.382387 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-69cdbd8b49-75d44: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-69cdbd8b49-75d44: 100
I0111 19:58:47.382673 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-7468b95b99-5rrqx' not found in {'psm-grpc-server-69cdbd8b49-75d44': 100} : Server psm-grpc-server-alt-7468b95b99-5rrqx did not receive RPCs.
I0111 19:58:48.383917 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:48.384191 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:53.430449 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7468b95b99-5rrqx: 92
psm-grpc-server-69cdbd8b49-75d44: 8
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7468b95b99-5rrqx: 92
psm-grpc-server-69cdbd8b49-75d44: 8
I0111 19:58:53.430794 139660000452608 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-69cdbd8b49-75d44' not found in ['psm-grpc-server-alt-7468b95b99-5rrqx'] : Unexpected server psm-grpc-server-69cdbd8b49-75d44 received RPCs.
I0111 19:58:54.432078 139660000452608 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-7468b95b99-5rrqx']
I0111 19:58:54.432406 139660000452608 grpc.py:75] [psm-grpc-client-bc5fc5bc6-mjjv6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 19:58:58.462263 139660000452608 xds_k8s_testcase.py:601] [psm-grpc-client-bc5fc5bc6-mjjv6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-7468b95b99-5rrqx: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-7468b95b99-5rrqx: 100
I0111 19:58:58.462595 139660000452608 xds_k8s_testcase.py:274] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.10_change_backend_service ---
I0111 19:58:58.462719 139660000452608 xds_k8s_testcase.py:687] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0111 19:58:58.685106 139660000452608 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-1953-20o1e"
I0111 19:58:59.281219 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003138689-60eb0fa6c29bd-6ff68c6f-72f373a2
I0111 19:59:12.473219 139660000452608 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-1953-20o1e"
I0111 19:59:12.586460 139660000452608 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-1953-20o1e"
I0111 19:59:12.995800 139660000452608 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-1953-20o1e"
I0111 19:59:13.370333 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003152999-60eb0fb468417-5dafcb4d-334ae9a4
I0111 19:59:15.640515 139660000452608 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-1953-20o1e"
I0111 19:59:16.058938 139660000452608 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-1953-20o1e"
I0111 19:59:16.717587 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003156064-60eb0fb75491b-5a002d0c-fac71bea
I0111 19:59:18.982602 139660000452608 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-1953-20o1e"
I0111 19:59:19.407786 139660000452608 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-1953-20o1e"
I0111 19:59:19.989174 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003159413-60eb0fba863e8-6c23bf6e-7a192969
I0111 19:59:45.983453 139660000452608 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-1953-20o1e"
I0111 19:59:46.640601 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003185988-60eb0fd3de47d-5dc40e73-8dfbbed0
I0111 20:00:16.912211 139660000452608 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-1953-20o1e"
I0111 20:00:17.413522 139660000452608 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-1953-20o1e"
I0111 20:00:17.975465 139660000452608 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003217570-60eb0ff1fc919-2ce2d0b9-1d934ce6
I0111 20:00:22.434183 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:00:32.613351 139660000452608 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:00:32.613661 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-1953-20o1e/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:00:33.190623 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:00:33.303453 139660000452608 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:00:33.303676 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-1953-20o1e
I0111 20:01:03.524804 139660000452608 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-1953-20o1e deleted
I0111 20:01:03.525126 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:01:13.698527 139660000452608 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:01:13.698828 139660000452608 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:01:13.815351 139660000452608 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:01:13.815662 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:01:14.644493 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:01:14.755231 139660000452608 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:01:14.755490 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1953-20o1e
I0111 20:02:15.002261 139660000452608 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-1953-20o1e deleted
I0111 20:02:15.002571 139660000452608 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:02:15.058662 139660000452608 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:02:15.115298 139660000452608 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-1953-20o1e/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
W0111 20:02:17.683681 139660000452608 http.py:163] Sleeping 0.85 seconds before retry 1 of 5 for request: POST https://iam.googleapis.com/v1/projects/grpc-testing/serviceAccounts/xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com:setIamPolicy?alt=json, after 502
I0111 20:02:18.956943 139660000452608 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:02:19.043891 139660000452608 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-1953-20o1e
I0111 20:02:19.079876 139660000452608 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:02:19.080252 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-6gyraz8u:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-1953-20o1e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-6gyraz8u%22;timeRange=2024-01-11T19:56:50.471180Z%2F2024-01-11T20:01:03.525094Z?project=grpc-testing
I0111 20:02:19.080415 139660000452608 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-snef9hve:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-1953-20o1e%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-snef9hve%22;timeRange=2024-01-11T19:54:58.990732Z%2F2024-01-11T20:02:15.002541Z?project=grpc-testing
[ OK ] ChangeBackendServiceTest.test_change_backend_service
I0111 20:02:19.080672 139660000452608 base_testcase.py:52] ----- TestCase __main__.ChangeBackendServiceTest.test_change_backend_service PASSED -----
----------------------------------------------------------------------
Ran 1 test in 530.426s
OK
+ for test in "${test_suites[@]}"
+ run_test custom_lb_test
+ local test_name=custom_lb_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test'
+ set -x
+ python -m tests.custom_lb_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:02:20.479438 139846197112832 xds_k8s_testcase.py:157] ----- Testing CustomLbTest -----
I0111 20:02:20.479620 139846197112832 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:02:20.479863 139846197112832 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:02:20.496662 139846197112832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:02:20.513510 139846197112832 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] CustomLbTest.test_custom_lb_config
I0111 20:02:20.518397 139846197112832 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2002-cg3w0
I0111 20:02:20.852132 139846197112832 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/custom_lb_test
I0111 20:02:21.881125 139846197112832 xds_k8s_testcase.py:663] Found unused xds port: 33446
I0111 20:02:21.881491 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0111 20:02:21.881592 139846197112832 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2002-cg3w0"
I0111 20:02:21.884764 139846197112832 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2002-cg3w0
type: GRPC
...
I0111 20:02:23.104360 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003341932-60eb10689669d-3d97c978-4191196f
I0111 20:02:25.461221 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.0_create_health_check ---
I0111 20:02:25.461428 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0111 20:02:25.461530 139846197112832 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2002-cg3w0"
I0111 20:02:25.470824 139846197112832 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2002-cg3w0
loadBalancingScheme: INTERNAL_SELF_MANAGED
localityLbPolicies:
- customPolicy:
data: '{ "foo": "bar" }'
name: test.ThisLoadBalancerDoesNotExist
- customPolicy:
data: '{ "rpcBehavior": "error-code-15" }'
name: test.RpcBehaviorLoadBalancer
name: psm-interop-backend-service-20240111-2002-cg3w0
protocol: GRPC
...
I0111 20:02:26.325885 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003345474-60eb106bf73a5-04d219e9-7cd47553
I0111 20:02:48.126820 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.1_create_backend_service ---
I0111 20:02:48.127070 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0111 20:02:48.127166 139846197112832 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2002-cg3w0": psm-grpc-server:33446 -> psm-interop-backend-service-20240111-2002-cg3w0
I0111 20:02:48.138430 139846197112832 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2002-cg3w0
hostRules:
- hosts:
- psm-grpc-server:33446
pathMatcher: psm-interop-path-matcher-20240111-2002-cg3w0
name: psm-interop-url-map-20240111-2002-cg3w0
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2002-cg3w0
name: psm-interop-path-matcher-20240111-2002-cg3w0
...
I0111 20:02:48.829525 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003368140-60eb108194f01-3d2a7cee-9d5285b6
I0111 20:02:51.216289 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.2_create_url_map ---
I0111 20:02:51.216522 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0111 20:02:51.216617 139846197112832 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2002-cg3w0 proxy "GRPC" to URL map psm-interop-url-map-20240111-2002-cg3w0
I0111 20:02:51.218715 139846197112832 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2002-cg3w0
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2002-cg3w0
validate_for_proxyless: true
...
I0111 20:02:51.782716 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003371220-60eb108484e67-9b1a7184-556228ca
I0111 20:02:54.049887 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.3_create_target_proxy ---
I0111 20:02:54.050159 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0111 20:02:54.050268 139846197112832 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2002-cg3w0" in network "default-vpc": 0.0.0.0:33446 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2002-cg3w0
I0111 20:02:54.052963 139846197112832 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2002-cg3w0
network: global/networks/default-vpc
portRange: 33446
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2002-cg3w0
...
I0111 20:02:55.107727 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003374055-60eb108738ea8-017e446e-62283dcc
I0111 20:03:08.007846 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.4_create_forwarding_rule ---
I0111 20:03:08.008091 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0111 20:03:08.008215 139846197112832 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2002-cg3w0: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:03:08.008456 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2002-cg3w0%22;timeRange=2024-01-11T20:03:08.008295Z%2F2024-01-11T20:33:08.008295Z?project=grpc-testing
I0111 20:03:08.013872 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2002-cg3w0
labels:
name: psm-interop-server-20240111-2002-cg3w0
owner: xds-k8s-interop-test
...
I0111 20:03:08.175436 139846197112832 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2002-cg3w0 created
I0111 20:03:08.178532 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2002-cg3w0-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:03:08.252398 139846197112832 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:03:08.252864 139846197112832 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2002-cg3w0
I0111 20:03:08.346837 139846197112832 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2002-cg3w0: neg_name=psm-interop-server-20240111-2002-cg3w0-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:03:08.347062 139846197112832 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2002-cg3w0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:03:09.261242 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:03:09.324512 139846197112832 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:03:09.334034 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2002-cg3w0
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-kn89amww
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2002-cg3w0-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:03:09.411838 139846197112832 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:03:09.412075 139846197112832 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:03:09.522282 139846197112832 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-758558bd4-5mr5j']
I0111 20:03:09.522584 139846197112832 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-758558bd4-5mr5j to start
I0111 20:03:12.781654 139846197112832 k8s_base_runner.py:820] Pod psm-grpc-server-758558bd4-5mr5j ready, IP: 10.120.2.173
I0111 20:03:12.781950 139846197112832 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-758558bd4-5mr5j to custom_lb_test/test_app_logs/psm-interop-server-20240111-2002-cg3w0_psm-grpc-server-758558bd4-5mr5j.log
I0111 20:03:12.782462 139846103852608 k8s_log_collector.py:70] Starting log collection thread 139846103852608 for psm-grpc-server-758558bd4-5mr5j
I0111 20:03:12.782639 139846197112832 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:03:19.195806 139846197112832 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:03:19.196253 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.5_start_test_server ---
I0111 20:03:19.196375 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0111 20:03:19.233335 139846197112832 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:03:19.357439 139846197112832 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2002-cg3w0-psm-grpc-server" in zone us-central1-a
I0111 20:03:19.357697 139846197112832 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2002-cg3w0: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2002-cg3w0-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2002-cg3w0-psm-grpc-server', zone='us-central1-a')}
I0111 20:03:19.362277 139846197112832 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2002-cg3w0-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:03:20.168581 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003399364-60eb109f5bf41-0389b081-d04ab39b
I0111 20:04:05.214567 139846197112832 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2002-cg3w0 to report all backends healthy: ['psm-interop-server-20240111-2002-cg3w0-psm-grpc-server']
I0111 20:04:05.405455 139846197112832 compute.py:494] Backend psm-interop-server-20240111-2002-cg3w0-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:04:09.410336 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.6_add_server_backends_to_backend_service ---
I0111 20:04:09.410570 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0111 20:04:09.410681 139846197112832 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2002-cg3w0: server_target=xds:///psm-grpc-server:33446 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:04:09.410934 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2002-cg3w0%22;timeRange=2024-01-11T20:04:09.410787Z%2F2024-01-11T20:34:09.410787Z?project=grpc-testing
I0111 20:04:09.413172 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2002-cg3w0
labels:
name: psm-interop-client-20240111-2002-cg3w0
owner: xds-k8s-interop-test
...
I0111 20:04:09.455238 139846197112832 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2002-cg3w0 created
I0111 20:04:09.455470 139846197112832 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2002-cg3w0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:04:10.129106 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2002-cg3w0
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:04:10.189777 139846197112832 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:04:10.198081 139846197112832 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2002-cg3w0
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-zkqpsbgt
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:33446"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:04:10.272607 139846197112832 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:04:10.272888 139846197112832 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:04:10.442100 139846197112832 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-f45fdd996-wnsn7']
I0111 20:04:10.442408 139846197112832 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-f45fdd996-wnsn7 to start
I0111 20:04:14.738801 139846197112832 k8s_base_runner.py:820] Pod psm-grpc-client-f45fdd996-wnsn7 ready, IP: 10.120.7.129
I0111 20:04:14.739075 139846197112832 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-f45fdd996-wnsn7 to custom_lb_test/test_app_logs/psm-interop-client-20240111-2002-cg3w0_psm-grpc-client-f45fdd996-wnsn7.log
I0111 20:04:14.739586 139846095459904 k8s_log_collector.py:70] Starting log collection thread 139846095459904 for psm-grpc-client-f45fdd996-wnsn7
I0111 20:04:14.739763 139846197112832 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:04:17.964606 139846197112832 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:04:17.965142 139846197112832 client_app.py:284] [psm-grpc-client-f45fdd996-wnsn7] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:04:18.069370 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:28.115473 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:38.160590 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:48.206417 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:04:58.251483 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:14.301072 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:39.361058 139846197112832 client_app.py:312] [psm-grpc-client-f45fdd996-wnsn7] xDS control plane channel:
I0111 20:05:41.396371 139846197112832 client_app.py:323] [psm-grpc-client-f45fdd996-wnsn7] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:05:41.396664 139846197112832 client_app.py:294] [psm-grpc-client-f45fdd996-wnsn7] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:05:41.396916 139846197112832 client_app.py:244] [psm-grpc-client-f45fdd996-wnsn7] Waiting to report a READY channel to xds:///psm-grpc-server:33446
I0111 20:05:41.429580 139846197112832 client_app.py:371] [psm-grpc-client-f45fdd996-wnsn7] Server channel:
I0111 20:05:41.462421 139846197112832 client_app.py:384] [psm-grpc-client-f45fdd996-wnsn7] Found subchannel in state READY:
I0111 20:05:41.462860 139846197112832 client_app.py:255] [psm-grpc-client-f45fdd996-wnsn7] Channel to xds:///psm-grpc-server:33446 transitioned to state READY:
I0111 20:05:41.463012 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.7_start_test_client ---
I0111 20:05:41.463092 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0111 20:05:41.463308 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), wait_for_ready=True, timeout=90)
I0111 20:05:41.498455 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.8_test_client_xds_config_exists ---
I0111 20:05:41.498718 139846197112832 xds_k8s_testcase.py:269] --- Starting subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0111 20:05:41.498947 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0111 20:05:51.543184 139846197112832 grpc.py:75] [psm-grpc-client-f45fdd996-wnsn7:8079] >> RPC LoadBalancerStatsService.GetClientAccumulatedStats(request=LoadBalancerAccumulatedStatsRequest({}), wait_for_ready=True, timeout=600)
I0111 20:05:51.579789 139846197112832 xds_k8s_testcase.py:398] [psm-grpc-client-f45fdd996-wnsn7] << Received accumulated stats difference. Expecting RPCs with status (15, DATA_LOSS) for method UNARY_CALL:
- method: UNARY_CALL
rpcs_started: 251
result:
(15, DATA_LOSS): 252
I0111 20:05:51.580506 139846197112832 xds_k8s_testcase.py:274] --- Finished subTest __main__.CustomLbTest.test_custom_lb_config.9_test_server_returned_configured_status_code ---
I0111 20:05:51.580660 139846197112832 xds_k8s_testcase.py:687] ----- TestMethod __main__.CustomLbTest.test_custom_lb_config teardown -----
I0111 20:05:51.806437 139846197112832 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2002-cg3w0"
I0111 20:05:52.552935 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003551810-60eb1130be3e5-af599ec5-4fc0a684
I0111 20:06:16.626568 139846197112832 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2002-cg3w0"
I0111 20:06:16.743684 139846197112832 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2002-cg3w0"
I0111 20:06:17.198181 139846197112832 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2002-cg3w0"
I0111 20:06:17.844416 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003577201-60eb1148f5440-9e68ded1-3d67c5ba
I0111 20:06:20.105163 139846197112832 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2002-cg3w0"
I0111 20:06:20.364686 139846197112832 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2002-cg3w0"
I0111 20:06:21.144745 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003580370-60eb114bfaf7c-f3900da1-dff61e6b
I0111 20:06:25.723691 139846197112832 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2002-cg3w0"
I0111 20:06:26.205460 139846197112832 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2002-cg3w0"
I0111 20:06:26.848117 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003586210-60eb11518cc78-fbd9abe1-3803ec1d
I0111 20:06:52.742841 139846197112832 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2002-cg3w0"
I0111 20:06:53.194561 139846197112832 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2002-cg3w0"
I0111 20:06:53.620371 139846197112832 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2002-cg3w0"
I0111 20:06:54.199699 139846197112832 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003613692-60eb116bc2371-69d84b05-95d6b990
I0111 20:06:58.568403 139846197112832 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:07:08.751038 139846197112832 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:07:08.751373 139846197112832 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2002-cg3w0/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:07:09.397170 139846197112832 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:07:09.509819 139846197112832 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:07:09.510140 139846197112832 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2002-cg3w0
I0111 20:07:39.693346 139846197112832 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2002-cg3w0 deleted
I0111 20:07:39.693661 139846197112832 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:07:49.930631 139846197112832 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:07:49.930931 139846197112832 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:07:50.054333 139846197112832 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:07:50.054588 139846197112832 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2002-cg3w0/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:07:50.599160 139846197112832 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:07:50.726630 139846197112832 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:07:50.726869 139846197112832 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2002-cg3w0
I0111 20:08:20.941802 139846197112832 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2002-cg3w0 deleted
I0111 20:08:20.942281 139846197112832 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:08:20.942540 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-zkqpsbgt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2002-cg3w0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-zkqpsbgt%22;timeRange=2024-01-11T20:04:09.410780Z%2F2024-01-11T20:07:39.693623Z?project=grpc-testing
I0111 20:08:20.942691 139846197112832 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-kn89amww:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2002-cg3w0%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-kn89amww%22;timeRange=2024-01-11T20:03:08.008288Z%2F2024-01-11T20:08:20.942112Z?project=grpc-testing
[ OK ] CustomLbTest.test_custom_lb_config
I0111 20:08:20.942970 139846197112832 base_testcase.py:52] ----- TestCase __main__.CustomLbTest.test_custom_lb_config PASSED -----
----------------------------------------------------------------------
Ran 1 test in 360.466s
OK
+ for test in "${test_suites[@]}"
+ run_test failover_test
+ local test_name=failover_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test'
+ set -x
+ python -m tests.failover_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:08:22.387956 139913641447424 xds_k8s_testcase.py:157] ----- Testing FailoverTest -----
I0111 20:08:22.388122 139913641447424 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:08:22.388356 139913641447424 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:08:22.406023 139913641447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:08:22.423303 139913641447424 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0111 20:08:22.428613 139913641447424 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2008-boazh
I0111 20:08:22.736891 139913641447424 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/failover_test
I0111 20:08:23.120027 139913641447424 xds_k8s_testcase.py:663] Found unused xds port: 6919
I0111 20:08:23.122310 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0111 20:08:23.122454 139913641447424 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2008-boazh"
I0111 20:08:23.125547 139913641447424 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2008-boazh
type: GRPC
...
I0111 20:08:23.759954 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003703169-60eb11c11731b-416258f6-6865d981
I0111 20:08:25.982944 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0111 20:08:25.983177 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0111 20:08:25.983278 139913641447424 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2008-boazh"
I0111 20:08:25.991434 139913641447424 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2008-boazh
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2008-boazh
protocol: GRPC
...
I0111 20:08:26.876192 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003705994-60eb11c3c8e4d-ba3342e5-f5545b95
I0111 20:09:05.874300 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0111 20:09:05.874537 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0111 20:09:05.874629 139913641447424 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2008-boazh": psm-grpc-server:6919 -> psm-interop-backend-service-20240111-2008-boazh
I0111 20:09:05.886891 139913641447424 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2008-boazh
hostRules:
- hosts:
- psm-grpc-server:6919
pathMatcher: psm-interop-path-matcher-20240111-2008-boazh
name: psm-interop-url-map-20240111-2008-boazh
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2008-boazh
name: psm-interop-path-matcher-20240111-2008-boazh
...
I0111 20:09:06.542138 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003745890-60eb11e9d5157-0fd663ea-4f83cadb
I0111 20:09:08.801229 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0111 20:09:08.801444 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0111 20:09:08.801552 139913641447424 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2008-boazh proxy "GRPC" to URL map psm-interop-url-map-20240111-2008-boazh
I0111 20:09:08.803504 139913641447424 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2008-boazh
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2008-boazh
validate_for_proxyless: true
...
I0111 20:09:09.283087 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003748805-60eb11ec9cc09-812eb62f-879a1c51
I0111 20:09:11.615242 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0111 20:09:11.615501 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0111 20:09:11.615630 139913641447424 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2008-boazh" in network "default-vpc": 0.0.0.0:6919 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2008-boazh
I0111 20:09:11.618214 139913641447424 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2008-boazh
network: global/networks/default-vpc
portRange: 6919
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2008-boazh
...
I0111 20:09:12.458112 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003751620-60eb11ef4bfc4-2168049e-fd4a833f
I0111 20:09:25.391936 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0111 20:09:25.392173 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0111 20:09:25.392290 139913641447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2008-boazh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0111 20:09:25.392578 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22;timeRange=2024-01-11T20:09:25.392380Z%2F2024-01-11T20:39:25.392380Z?project=grpc-testing
I0111 20:09:25.397961 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2008-boazh
labels:
name: psm-interop-server-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:09:25.536939 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2008-boazh created
I0111 20:09:25.539982 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2008-boazh-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:09:25.614156 139913641447424 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:09:25.614384 139913641447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2008-boazh
I0111 20:09:25.706863 139913641447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2008-boazh: neg_name=psm-interop-server-20240111-2008-boazh-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:09:25.707097 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:09:26.505964 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:09:26.576163 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:09:26.585172 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2008-boazh
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-4rktopzs
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2008-boazh-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:09:26.660861 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:09:26.661137 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0111 20:09:26.815884 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:09:26.816328 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-6l52f to start
I0111 20:09:29.005968 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-6l52f ready, IP: 10.120.0.71
I0111 20:09:29.006224 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-6l52f to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-6l52f.log
I0111 20:09:29.006721 139913548260928 k8s_log_collector.py:70] Starting log collection thread 139913548260928 for psm-grpc-server-76f54665-6l52f
I0111 20:09:29.006996 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-hjgdh to start
I0111 20:09:30.225463 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-hjgdh ready, IP: 10.120.3.103
I0111 20:09:30.225731 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-hjgdh to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-hjgdh.log
I0111 20:09:30.226195 139913539868224 k8s_log_collector.py:70] Starting log collection thread 139913539868224 for psm-grpc-server-76f54665-hjgdh
I0111 20:09:30.226339 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-76f54665-zsx45 to start
I0111 20:09:30.400224 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-76f54665-zsx45 ready, IP: 10.120.2.174
I0111 20:09:30.400607 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-76f54665-zsx45 to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-76f54665-zsx45.log
I0111 20:09:30.401073 139913458611776 k8s_log_collector.py:70] Starting log collection thread 139913458611776 for psm-grpc-server-76f54665-zsx45
I0111 20:09:30.401249 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0111 20:09:36.738187 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-server has 3 replicas available
I0111 20:09:36.738750 139913641447424 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-2008-boazh: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:09:36.739004 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22;timeRange=2024-01-11T20:09:36.738882Z%2F2024-01-11T20:39:36.738882Z?project=grpc-testing
I0111 20:09:36.740842 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2008-boazh
labels:
name: psm-interop-server-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:09:36.990764 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2008-boazh created
I0111 20:09:36.993801 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:09:37.121321 139913641447424 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 20:09:37.121567 139913641447424 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-2008-boazh
I0111 20:09:37.280402 139913641447424 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-2008-boazh: neg_name=psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt, port=8080, zones=['us-west1-b']
I0111 20:09:37.280618 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:09:37.931873 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:09:38.023608 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 20:09:38.031785 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2008-boazh
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-rjq9y3ce
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:09:38.137392 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 20:09:38.137653 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 20:09:38.316219 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:09:38.316516 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-6c6cc65f86-v4lv8 to start
I0111 20:09:44.951710 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-server-alt-6c6cc65f86-v4lv8 ready, IP: 10.108.6.100
I0111 20:09:44.951983 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-6c6cc65f86-v4lv8 to failover_test/test_app_logs/psm-interop-server-20240111-2008-boazh_psm-grpc-server-alt-6c6cc65f86-v4lv8.log
I0111 20:09:44.952458 139913450219072 k8s_log_collector.py:70] Starting log collection thread 139913450219072 for psm-grpc-server-alt-6c6cc65f86-v4lv8
I0111 20:09:44.952631 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 20:09:49.554591 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 20:09:49.555037 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0111 20:09:49.555170 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0111 20:09:49.593019 139913641447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:09:49.890694 139913641447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2008-boazh-psm-grpc-server" in zone us-central1-a
I0111 20:09:49.890915 139913641447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2008-boazh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server', zone='us-central1-a')}
I0111 20:09:49.895409 139913641447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server
maxRatePerEndpoint: 100
...
I0111 20:09:51.177198 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003789898-60eb1213cd43e-d9c184c7-5b75c0b3
I0111 20:10:40.993617 139913641447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2008-boazh to report all backends healthy: ['psm-interop-server-20240111-2008-boazh-psm-grpc-server']
I0111 20:10:41.182751 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:10:45.259872 139913641447424 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:10:45.734279 139913641447424 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt" in zone us-west1-b
I0111 20:10:45.734519 139913641447424 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2008-boazh: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2008-boazh-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server', zone='us-central1-a')}
I0111 20:10:45.739555 139913641447424 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2008-boazh-psm-grpc-server
maxRatePerEndpoint: 100
...
I0111 20:10:47.732389 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705003845741-60eb12490ed43-b8015dee-7a3ac74b
I0111 20:11:39.755125 139913641447424 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2008-boazh to report all backends healthy: ['psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt', 'psm-interop-server-20240111-2008-boazh-psm-grpc-server']
I0111 20:11:40.637006 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server-alt in zone us-west1-b reported healthy
I0111 20:11:45.216178 139913641447424 compute.py:494] Backend psm-interop-server-20240111-2008-boazh-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:11:49.221073 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0111 20:11:49.221302 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0111 20:11:49.221412 139913641447424 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2008-boazh: server_target=xds:///psm-grpc-server:6919 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:11:49.221659 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2008-boazh%22;timeRange=2024-01-11T20:11:49.221493Z%2F2024-01-11T20:41:49.221493Z?project=grpc-testing
I0111 20:11:49.223608 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2008-boazh
labels:
name: psm-interop-client-20240111-2008-boazh
owner: xds-k8s-interop-test
...
I0111 20:11:49.368944 139913641447424 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2008-boazh created
I0111 20:11:49.369181 139913641447424 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2008-boazh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:11:49.956958 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2008-boazh
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:11:50.015135 139913641447424 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:11:50.023332 139913641447424 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2008-boazh
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-gx2jhwbf
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:6919"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:11:50.107287 139913641447424 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:11:50.107581 139913641447424 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:11:50.210240 139913641447424 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-6b767fbc77-vsz7v']
I0111 20:11:50.210525 139913641447424 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-6b767fbc77-vsz7v to start
I0111 20:11:53.439570 139913641447424 k8s_base_runner.py:820] Pod psm-grpc-client-6b767fbc77-vsz7v ready, IP: 10.120.8.64
I0111 20:11:53.439865 139913641447424 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-6b767fbc77-vsz7v to failover_test/test_app_logs/psm-interop-client-20240111-2008-boazh_psm-grpc-client-6b767fbc77-vsz7v.log
I0111 20:11:53.440364 139913441826368 k8s_log_collector.py:70] Starting log collection thread 139913441826368 for psm-grpc-client-6b767fbc77-vsz7v
I0111 20:11:53.440528 139913641447424 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:11:54.576689 139913641447424 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:11:54.577218 139913641447424 client_app.py:284] [psm-grpc-client-6b767fbc77-vsz7v] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:11:54.680208 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:04.724061 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:14.768380 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:24.812790 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:34.857550 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:12:50.906437 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:13:15.966430 139913641447424 client_app.py:312] [psm-grpc-client-6b767fbc77-vsz7v] xDS control plane channel:
I0111 20:13:18.001230 139913641447424 client_app.py:323] [psm-grpc-client-6b767fbc77-vsz7v] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:13:18.001528 139913641447424 client_app.py:294] [psm-grpc-client-6b767fbc77-vsz7v] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:13:18.001826 139913641447424 client_app.py:244] [psm-grpc-client-6b767fbc77-vsz7v] Waiting to report a READY channel to xds:///psm-grpc-server:6919
I0111 20:13:18.034170 139913641447424 client_app.py:371] [psm-grpc-client-6b767fbc77-vsz7v] Server channel:
I0111 20:13:18.066734 139913641447424 client_app.py:384] [psm-grpc-client-6b767fbc77-vsz7v] Found subchannel in state READY:
I0111 20:13:18.067214 139913641447424 client_app.py:255] [psm-grpc-client-6b767fbc77-vsz7v] Channel to xds:///psm-grpc-server:6919 transitioned to state READY:
I0111 20:13:18.067361 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0111 20:13:18.067438 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0111 20:13:18.067713 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:18.102243 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.08_test_client_xds_config_exists ---
I0111 20:13:18.102484 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0111 20:13:18.102802 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:18.103056 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:22.121807 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 34
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 34
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 33
I0111 20:13:22.122264 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.09_primary_locality_receives_requests ---
I0111 20:13:22.122374 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0111 20:13:22.122452 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-6l52f] >> Setting health status to NOT_SERVING
I0111 20:13:22.122793 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:13:22.219638 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:22.251611 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-6l52f] << Health status status: NOT_SERVING
I0111 20:13:22.252102 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:22.252297 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:26.281747 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 34
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 33
psm-grpc-server-76f54665-hjgdh: 34
I0111 20:13:26.282323 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:27.283626 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:27.283970 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:31.282632 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 34
psm-grpc-server-76f54665-hjgdh: 33
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 33
psm-grpc-server-76f54665-zsx45: 34
psm-grpc-server-76f54665-hjgdh: 33
I0111 20:13:31.283039 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:32.284381 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:32.284735 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:36.281849 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 4
psm-grpc-server-76f54665-zsx45: 48
psm-grpc-server-76f54665-hjgdh: 48
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 4
psm-grpc-server-76f54665-zsx45: 48
psm-grpc-server-76f54665-hjgdh: 48
I0111 20:13:36.282249 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45'] : Unexpected server psm-grpc-server-76f54665-6l52f received RPCs.
I0111 20:13:37.283644 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:13:37.284029 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:41.281406 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:41.281740 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.10_secondary_locality_receives_no_requests_on_partial_primary_failure ---
I0111 20:13:41.281833 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0111 20:13:41.281903 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-hjgdh] >> Setting health status to NOT_SERVING
I0111 20:13:41.282249 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:13:41.378674 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:13:41.410757 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-hjgdh] << Health status status: NOT_SERVING
I0111 20:13:41.411232 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:41.411411 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:45.442096 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:45.442679 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-6c6cc65f86-v4lv8' not found in {'psm-grpc-server-76f54665-zsx45': 50, 'psm-grpc-server-76f54665-hjgdh': 50} : Server psm-grpc-server-alt-6c6cc65f86-v4lv8 did not receive RPCs.
I0111 20:13:46.444025 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:46.444355 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:50.441465 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 50
psm-grpc-server-76f54665-hjgdh: 50
I0111 20:13:50.441771 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-alt-6c6cc65f86-v4lv8' not found in {'psm-grpc-server-76f54665-zsx45': 50, 'psm-grpc-server-76f54665-hjgdh': 50} : Server psm-grpc-server-alt-6c6cc65f86-v4lv8 did not receive RPCs.
I0111 20:13:51.443064 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:51.443352 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:13:55.441804 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 60
psm-grpc-server-76f54665-hjgdh: 16
psm-grpc-server-alt-6c6cc65f86-v4lv8: 24
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 60
psm-grpc-server-76f54665-hjgdh: 16
psm-grpc-server-alt-6c6cc65f86-v4lv8: 24
I0111 20:13:55.442188 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-hjgdh' not found in ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-hjgdh received RPCs.
I0111 20:13:56.443488 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-zsx45', 'psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:13:56.443871 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:00.442048 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
I0111 20:14:00.442391 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.11_gentle_failover ---
I0111 20:14:00.442486 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0111 20:14:00.442556 139913641447424 server_app.py:98] [psm-grpc-server-76f54665-zsx45] >> Setting health status to NOT_SERVING
I0111 20:14:00.442892 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC XdsUpdateHealthService.SetNotServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:00.539745 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:00.572016 139913641447424 server_app.py:102] [psm-grpc-server-76f54665-zsx45] << Health status status: NOT_SERVING
I0111 20:14:00.572509 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:00.572736 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:04.633217 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 68
psm-grpc-server-alt-6c6cc65f86-v4lv8: 32
I0111 20:14:04.633792 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-zsx45' not found in ['psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-zsx45 received RPCs.
I0111 20:14:05.635100 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:05.635426 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:09.673938 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-zsx45: 15
psm-grpc-server-alt-6c6cc65f86-v4lv8: 85
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-zsx45: 15
psm-grpc-server-alt-6c6cc65f86-v4lv8: 85
I0111 20:14:09.674326 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-zsx45' not found in ['psm-grpc-server-alt-6c6cc65f86-v4lv8'] : Unexpected server psm-grpc-server-76f54665-zsx45 received RPCs.
I0111 20:14:10.675648 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-alt-6c6cc65f86-v4lv8']
I0111 20:14:10.676008 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:14.712773 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
I0111 20:14:14.713163 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.12_secondary_locality_receives_requests_on_primary_failure ---
I0111 20:14:14.713267 139913641447424 xds_k8s_testcase.py:269] --- Starting subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0111 20:14:14.713340 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-6l52f] >> Setting health status to SERVING
I0111 20:14:14.713447 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.747150 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-6l52f:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.779174 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-6l52f] << Health status status: SERVING
I0111 20:14:14.779399 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-hjgdh] >> Setting health status to SERVING
I0111 20:14:14.779556 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.811865 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-hjgdh:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.843859 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-hjgdh] << Health status status: SERVING
I0111 20:14:14.844110 139913641447424 server_app.py:89] [psm-grpc-server-76f54665-zsx45] >> Setting health status to SERVING
I0111 20:14:14.844231 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC XdsUpdateHealthService.SetServing(request=Empty({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.878186 139913641447424 grpc.py:75] [psm-grpc-server-76f54665-zsx45:8080] >> RPC Health.Check(request=HealthCheckRequest({}), timeout=90, wait_for_ready=True)
I0111 20:14:14.910275 139913641447424 server_app.py:91] [psm-grpc-server-76f54665-zsx45] << Health status status: SERVING
I0111 20:14:14.910741 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:14.910954 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:18.953256 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-6c6cc65f86-v4lv8: 100
I0111 20:14:18.953824 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-6l52f' not found in {'psm-grpc-server-alt-6c6cc65f86-v4lv8': 100} : Server psm-grpc-server-76f54665-6l52f did not receive RPCs.
I0111 20:14:19.955106 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:19.955458 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:23.961477 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 21
psm-grpc-server-alt-6c6cc65f86-v4lv8: 79
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 21
psm-grpc-server-alt-6c6cc65f86-v4lv8: 79
I0111 20:14:23.961865 139913641447424 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-76f54665-hjgdh' not found in {'psm-grpc-server-76f54665-6l52f': 21, 'psm-grpc-server-alt-6c6cc65f86-v4lv8': 79} : Server psm-grpc-server-76f54665-hjgdh did not receive RPCs.
I0111 20:14:24.963189 139913641447424 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-76f54665-6l52f', 'psm-grpc-server-76f54665-hjgdh', 'psm-grpc-server-76f54665-zsx45']
I0111 20:14:24.963607 139913641447424 grpc.py:75] [psm-grpc-client-6b767fbc77-vsz7v:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:14:28.962077 139913641447424 xds_k8s_testcase.py:601] [psm-grpc-client-6b767fbc77-vsz7v] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-76f54665-6l52f: 41
psm-grpc-server-76f54665-zsx45: 40
psm-grpc-server-76f54665-hjgdh: 19
rpcs_by_method:
UnaryCall:
psm-grpc-server-76f54665-6l52f: 41
psm-grpc-server-76f54665-zsx45: 40
psm-grpc-server-76f54665-hjgdh: 19
I0111 20:14:28.962437 139913641447424 xds_k8s_testcase.py:274] --- Finished subTest __main__.FailoverTest.test_failover.13_traffic_resumes_to_healthy_backends ---
I0111 20:14:28.962561 139913641447424 xds_k8s_testcase.py:687] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0111 20:14:29.223731 139913641447424 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2008-boazh"
I0111 20:14:29.937295 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004069228-60eb131e311bd-f73dc2dc-74af7019
I0111 20:14:43.033998 139913641447424 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2008-boazh"
I0111 20:14:43.141362 139913641447424 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2008-boazh"
I0111 20:14:43.597512 139913641447424 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2008-boazh"
I0111 20:14:44.139784 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004083600-60eb132be5d8d-84bca90d-4d56dbce
I0111 20:14:46.400274 139913641447424 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2008-boazh"
I0111 20:14:46.738552 139913641447424 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2008-boazh"
I0111 20:14:47.456656 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004086743-60eb132ee5327-d77472f9-c7e90255
I0111 20:14:49.844459 139913641447424 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2008-boazh"
I0111 20:14:50.294652 139913641447424 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2008-boazh"
I0111 20:14:50.891716 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004090300-60eb13324985b-2483acc5-f00b137c
I0111 20:15:27.600957 139913641447424 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2008-boazh"
I0111 20:15:27.942696 139913641447424 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2008-boazh"
I0111 20:15:28.400301 139913641447424 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2008-boazh"
I0111 20:15:28.865722 139913641447424 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004128476-60eb1356b1ba5-9b3e8adf-fd8bdb5c
I0111 20:15:33.329291 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:15:43.526966 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:15:43.527270 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2008-boazh/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:15:44.061680 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:15:44.178549 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:15:44.178780 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2008-boazh
I0111 20:16:14.361317 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2008-boazh deleted
I0111 20:16:14.361594 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:16:24.535983 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:16:24.536315 139913641447424 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:16:24.650049 139913641447424 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:16:24.650289 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:16:25.266630 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:16:25.389813 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:16:25.390047 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2008-boazh
I0111 20:18:55.918070 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2008-boazh deleted
I0111 20:18:55.918290 139913641447424 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:19:06.362561 139913641447424 k8s_base_runner.py:723] Deployment psm-grpc-server-alt deleted
I0111 20:19:06.362869 139913641447424 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:19:06.550542 139913641447424 k8s_base_runner.py:739] Service psm-grpc-server-alt deleted
I0111 20:19:06.550800 139913641447424 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2008-boazh/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:19:07.112985 139913641447424 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:19:07.296326 139913641447424 k8s_base_runner.py:756] Service account psm-grpc-server-alt deleted
I0111 20:19:07.296561 139913641447424 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2008-boazh
I0111 20:21:07.905254 139913641447424 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2008-boazh deleted
I0111 20:21:07.905699 139913641447424 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:21:07.905927 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-gx2jhwbf:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2008-boazh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-gx2jhwbf%22;timeRange=2024-01-11T20:11:49.221485Z%2F2024-01-11T20:16:14.361563Z?project=grpc-testing
I0111 20:21:07.906091 139913641447424 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-4rktopzs:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2008-boazh%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-4rktopzs%22;timeRange=2024-01-11T20:09:25.392372Z%2F2024-01-11T20:18:55.918259Z?project=grpc-testing
[ OK ] FailoverTest.test_failover
I0111 20:21:07.906363 139913641447424 base_testcase.py:52] ----- TestCase __main__.FailoverTest.test_failover PASSED -----
----------------------------------------------------------------------
Ran 1 test in 765.521s
OK
+ for test in "${test_suites[@]}"
+ run_test outlier_detection_test
+ local test_name=outlier_detection_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test'
+ set -x
+ python -m tests.outlier_detection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:21:09.368448 140660168441856 xds_k8s_testcase.py:157] ----- Testing OutlierDetectionTest -----
I0111 20:21:09.368624 140660168441856 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:21:09.368925 140660168441856 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:21:09.386436 140660168441856 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:21:09.403961 140660168441856 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0111 20:21:09.408847 140660168441856 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2021-7a5zc
I0111 20:21:09.713453 140660168441856 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/outlier_detection_test
I0111 20:21:10.025703 140660168441856 xds_k8s_testcase.py:663] Found unused xds port: 58833
I0111 20:21:10.026110 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0111 20:21:10.026195 140660168441856 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2021-7a5zc"
I0111 20:21:10.029386 140660168441856 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2021-7a5zc
type: GRPC
...
I0111 20:21:10.795434 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004470091-60eb149c7be36-ab19e17e-ba69098e
I0111 20:21:13.144676 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0111 20:21:13.144854 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0111 20:21:13.144926 140660168441856 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2021-7a5zc"
I0111 20:21:13.153614 140660168441856 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2021-7a5zc
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2021-7a5zc
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...
I0111 20:21:13.803114 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004473155-60eb149f68097-ee1e17fd-8daefb97
I0111 20:22:03.405103 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0111 20:22:03.405283 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0111 20:22:03.405356 140660168441856 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2021-7a5zc": psm-grpc-server:58833 -> psm-interop-backend-service-20240111-2021-7a5zc
I0111 20:22:03.416995 140660168441856 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2021-7a5zc
hostRules:
- hosts:
- psm-grpc-server:58833
pathMatcher: psm-interop-path-matcher-20240111-2021-7a5zc
name: psm-interop-url-map-20240111-2021-7a5zc
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2021-7a5zc
name: psm-interop-path-matcher-20240111-2021-7a5zc
...
I0111 20:22:04.110036 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004523418-60eb14cf574da-e90db041-8b4dc542
I0111 20:22:06.475903 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0111 20:22:06.476128 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0111 20:22:06.476220 140660168441856 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2021-7a5zc proxy "GRPC" to URL map psm-interop-url-map-20240111-2021-7a5zc
I0111 20:22:06.478308 140660168441856 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2021-7a5zc
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2021-7a5zc
validate_for_proxyless: true
...
I0111 20:22:07.033910 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004526480-60eb14d242c44-d71443ed-dc74897c
I0111 20:22:09.435275 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0111 20:22:09.435516 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0111 20:22:09.435667 140660168441856 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2021-7a5zc" in network "default-vpc": 0.0.0.0:58833 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2021-7a5zc
I0111 20:22:09.438304 140660168441856 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2021-7a5zc
network: global/networks/default-vpc
portRange: 58833
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2021-7a5zc
...
I0111 20:22:10.330671 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004529440-60eb14d51563f-d48da57a-1f010279
I0111 20:22:42.813504 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0111 20:22:42.813761 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0111 20:22:42.813870 140660168441856 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2021-7a5zc: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0111 20:22:42.814109 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2021-7a5zc%22;timeRange=2024-01-11T20:22:42.813951Z%2F2024-01-11T20:52:42.813951Z?project=grpc-testing
I0111 20:22:42.819211 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2021-7a5zc
labels:
name: psm-interop-server-20240111-2021-7a5zc
owner: xds-k8s-interop-test
...
I0111 20:22:42.972466 140660168441856 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2021-7a5zc created
I0111 20:22:42.975427 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2021-7a5zc-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:22:43.046399 140660168441856 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:22:43.046630 140660168441856 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2021-7a5zc
I0111 20:22:43.145320 140660168441856 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2021-7a5zc: neg_name=psm-interop-server-20240111-2021-7a5zc-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:22:43.145573 140660168441856 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2021-7a5zc/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:22:43.860912 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:22:43.919785 140660168441856 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:22:43.928897 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2021-7a5zc
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-13xkr660
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:canonical-v1.56
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2021-7a5zc-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:22:44.003890 140660168441856 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:22:44.004150 140660168441856 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0111 20:22:44.186730 140660168441856 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:22:44.187257 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-bp577 to start
I0111 20:22:47.496472 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-bp577 ready, IP: 10.120.4.250
I0111 20:22:47.496746 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-bp577 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-bp577.log
I0111 20:22:47.497256 140660075652672 k8s_log_collector.py:70] Starting log collection thread 140660075652672 for psm-grpc-server-6dc84f789b-bp577
I0111 20:22:47.497416 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-cb857 to start
I0111 20:22:47.676731 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-cb857 ready, IP: 10.120.3.106
I0111 20:22:47.677074 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-cb857 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-cb857.log
I0111 20:22:47.677571 140660067259968 k8s_log_collector.py:70] Starting log collection thread 140660067259968 for psm-grpc-server-6dc84f789b-cb857
I0111 20:22:47.677767 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-gn4zb to start
I0111 20:22:47.779759 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-gn4zb ready, IP: 10.120.2.176
I0111 20:22:47.780034 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-gn4zb to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-gn4zb.log
I0111 20:22:47.780475 140660058867264 k8s_log_collector.py:70] Starting log collection thread 140660058867264 for psm-grpc-server-6dc84f789b-gn4zb
I0111 20:22:47.780663 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-jz7n8 to start
I0111 20:22:47.959130 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-jz7n8 ready, IP: 10.120.0.73
I0111 20:22:47.959456 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-jz7n8 to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-jz7n8.log
I0111 20:22:47.959986 140659843397184 k8s_log_collector.py:70] Starting log collection thread 140659843397184 for psm-grpc-server-6dc84f789b-jz7n8
I0111 20:22:47.960211 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6dc84f789b-kpttb to start
I0111 20:22:48.072962 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-server-6dc84f789b-kpttb ready, IP: 10.120.8.65
I0111 20:22:48.073245 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6dc84f789b-kpttb to outlier_detection_test/test_app_logs/psm-interop-server-20240111-2021-7a5zc_psm-grpc-server-6dc84f789b-kpttb.log
I0111 20:22:48.073747 140659835004480 k8s_log_collector.py:70] Starting log collection thread 140659835004480 for psm-grpc-server-6dc84f789b-kpttb
I0111 20:22:48.074020 140660168441856 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0111 20:22:56.495351 140660168441856 k8s_base_runner.py:788] Deployment psm-grpc-server has 5 replicas available
I0111 20:22:56.496175 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0111 20:22:56.496340 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0111 20:22:56.533211 140660168441856 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:22:56.638636 140660168441856 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2021-7a5zc-psm-grpc-server" in zone us-central1-a
I0111 20:22:56.638876 140660168441856 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2021-7a5zc: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2021-7a5zc-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2021-7a5zc-psm-grpc-server', zone='us-central1-a')}
I0111 20:22:56.643366 140660168441856 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2021-7a5zc-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:22:57.738445 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004576647-60eb15021a77d-bc0d602d-0ffdd92b
I0111 20:23:43.134381 140660168441856 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2021-7a5zc to report all backends healthy: ['psm-interop-server-20240111-2021-7a5zc-psm-grpc-server']
I0111 20:24:09.021908 140660168441856 compute.py:494] Backend psm-interop-server-20240111-2021-7a5zc-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:24:13.026675 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0111 20:24:13.026959 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0111 20:24:13.027091 140660168441856 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2021-7a5zc: server_target=xds:///psm-grpc-server:58833 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0111 20:24:13.027304 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2021-7a5zc%22;timeRange=2024-01-11T20:24:13.027174Z%2F2024-01-11T20:54:13.027174Z?project=grpc-testing
I0111 20:24:13.029269 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2021-7a5zc
labels:
name: psm-interop-client-20240111-2021-7a5zc
owner: xds-k8s-interop-test
...
I0111 20:24:13.070690 140660168441856 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2021-7a5zc created
I0111 20:24:13.070942 140660168441856 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2021-7a5zc/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:24:13.682441 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2021-7a5zc
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:24:13.744079 140660168441856 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:24:13.752298 140660168441856 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2021-7a5zc
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-vv7smh7f
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:58833"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:24:13.826289 140660168441856 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:24:13.826526 140660168441856 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:24:13.938238 140660168441856 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-75ddf569df-nkdrp']
I0111 20:24:13.938524 140660168441856 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-75ddf569df-nkdrp to start
I0111 20:24:17.182074 140660168441856 k8s_base_runner.py:820] Pod psm-grpc-client-75ddf569df-nkdrp ready, IP: 10.120.5.4
I0111 20:24:17.182358 140660168441856 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-75ddf569df-nkdrp to outlier_detection_test/test_app_logs/psm-interop-client-20240111-2021-7a5zc_psm-grpc-client-75ddf569df-nkdrp.log
I0111 20:24:17.182831 140659826611776 k8s_log_collector.py:70] Starting log collection thread 140659826611776 for psm-grpc-client-75ddf569df-nkdrp
I0111 20:24:17.183141 140660168441856 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:24:18.398954 140660168441856 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:24:18.399569 140660168441856 client_app.py:284] [psm-grpc-client-75ddf569df-nkdrp] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:24:18.502339 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:28.546621 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:38.591340 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:48.635727 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:24:58.680468 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:14.729544 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:39.788998 140660168441856 client_app.py:312] [psm-grpc-client-75ddf569df-nkdrp] xDS control plane channel:
I0111 20:25:41.823925 140660168441856 client_app.py:323] [psm-grpc-client-75ddf569df-nkdrp] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:25:41.824228 140660168441856 client_app.py:294] [psm-grpc-client-75ddf569df-nkdrp] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:25:41.824503 140660168441856 client_app.py:244] [psm-grpc-client-75ddf569df-nkdrp] Waiting to report a READY channel to xds:///psm-grpc-server:58833
I0111 20:25:41.856519 140660168441856 client_app.py:371] [psm-grpc-client-75ddf569df-nkdrp] Server channel:
I0111 20:25:41.888895 140660168441856 client_app.py:384] [psm-grpc-client-75ddf569df-nkdrp] Found subchannel in state READY:
I0111 20:25:41.889298 140660168441856 client_app.py:255] [psm-grpc-client-75ddf569df-nkdrp] Channel to xds:///psm-grpc-server:58833 transitioned to state READY:
I0111 20:25:41.889430 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0111 20:25:41.889507 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0111 20:25:41.889702 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:25:41.924002 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.08_test_client_xds_config_exists ---
I0111 20:25:41.924207 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0111 20:25:41.924525 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:41.924742 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:42.958028 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
I0111 20:25:42.958505 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.09_test_servers_received_rpcs_from_test_client ---
I0111 20:25:42.958621 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0111 20:25:42.958865 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL'], 'metadata': [{'type': 'UNARY_CALL', 'key': 'rpc-behavior', 'value': 'hostname=psm-grpc-server-6dc84f789b-bp577 error-code-2'}]}), timeout=5, wait_for_ready=True)
I0111 20:25:42.991240 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:42.991556 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:44.018147 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 10
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 22
psm-grpc-server-6dc84f789b-cb857: 23
psm-grpc-server-6dc84f789b-jz7n8: 22
psm-grpc-server-6dc84f789b-gn4zb: 23
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 22
psm-grpc-server-6dc84f789b-cb857: 23
psm-grpc-server-6dc84f789b-jz7n8: 22
psm-grpc-server-6dc84f789b-gn4zb: 23
I0111 20:25:44.018746 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 10 not less than or equal to 0 : Expected all RPCs to succeed: 10 of 100 failed.
I0111 20:25:45.020087 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:45.020440 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:46.047940 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:46.048268 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.10_chosen_server_removed_by_outlier_detection ---
I0111 20:25:46.048366 140660168441856 xds_k8s_testcase.py:269] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0111 20:25:46.048505 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC XdsUpdateClientConfigureService.Configure(request=ClientConfigureRequest({'types': ['UNARY_CALL']}), timeout=5, wait_for_ready=True)
I0111 20:25:46.080642 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:46.080999 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:47.108057 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:47.108638 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:48.110038 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:48.110416 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:49.140868 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:49.141255 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:50.142601 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:50.142951 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:51.178625 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:51.179062 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:52.180418 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:52.180761 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:53.208257 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:53.208636 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:54.209966 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:54.210358 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:55.238577 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:55.238999 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:56.240319 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:56.240656 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:57.267826 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:57.268180 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:25:58.269450 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:25:58.269842 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:25:59.297893 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:25:59.298295 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:00.299602 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:00.299935 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:01.328274 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:01.328645 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:02.329962 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:02.330323 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:03.357577 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:03.358020 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:04.359321 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:04.359689 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:05.388402 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:05.388811 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:06.390127 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:06.390504 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:07.418077 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:07.418446 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:08.419702 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:08.420060 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:09.447746 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:09.448133 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:10.449466 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:10.449845 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:11.477724 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:11.478088 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:12.479361 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:12.479707 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:13.508231 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 25
psm-grpc-server-6dc84f789b-cb857: 25
psm-grpc-server-6dc84f789b-jz7n8: 25
psm-grpc-server-6dc84f789b-gn4zb: 25
I0111 20:26:13.508622 140660168441856 retryers.py:206] Retrying framework.xds_k8s_testcase.XdsKubernetesBaseTestCase._assertRpcsEventuallyGoToGivenServers in 1.0 seconds as it raised AssertionError: 'psm-grpc-server-6dc84f789b-bp577' not found in {'psm-grpc-server-6dc84f789b-kpttb': 25, 'psm-grpc-server-6dc84f789b-cb857': 25, 'psm-grpc-server-6dc84f789b-jz7n8': 25, 'psm-grpc-server-6dc84f789b-gn4zb': 25} : Server psm-grpc-server-6dc84f789b-bp577 did not receive RPCs.
I0111 20:26:14.509884 140660168441856 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-6dc84f789b-bp577', 'psm-grpc-server-6dc84f789b-cb857', 'psm-grpc-server-6dc84f789b-gn4zb', 'psm-grpc-server-6dc84f789b-jz7n8', 'psm-grpc-server-6dc84f789b-kpttb']
I0111 20:26:14.510233 140660168441856 grpc.py:75] [psm-grpc-client-75ddf569df-nkdrp:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:26:15.538199 140660168441856 xds_k8s_testcase.py:601] [psm-grpc-client-75ddf569df-nkdrp] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
rpcs_by_method:
UnaryCall:
psm-grpc-server-6dc84f789b-kpttb: 20
psm-grpc-server-6dc84f789b-cb857: 20
psm-grpc-server-6dc84f789b-bp577: 20
psm-grpc-server-6dc84f789b-jz7n8: 20
psm-grpc-server-6dc84f789b-gn4zb: 20
I0111 20:26:15.538533 140660168441856 xds_k8s_testcase.py:274] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.11_ejected_server_returned_after_failures_stopped ---
I0111 20:26:15.538653 140660168441856 xds_k8s_testcase.py:687] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0111 20:26:15.836634 140660168441856 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2021-7a5zc"
I0111 20:26:16.699855 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004775840-60eb15c011beb-5772243e-ee9f751c
I0111 20:26:44.995255 140660168441856 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2021-7a5zc"
I0111 20:26:45.087317 140660168441856 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2021-7a5zc"
I0111 20:26:45.515568 140660168441856 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2021-7a5zc"
I0111 20:26:45.966248 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004805518-60eb15dc5f5ac-9711354f-e902c5e5
I0111 20:26:48.230611 140660168441856 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2021-7a5zc"
I0111 20:26:48.587856 140660168441856 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2021-7a5zc"
I0111 20:26:49.325899 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004808593-60eb15df4ded8-5e8b0943-4a79d570
I0111 20:26:51.643544 140660168441856 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2021-7a5zc"
I0111 20:26:52.082852 140660168441856 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2021-7a5zc"
I0111 20:26:52.713625 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004812088-60eb15e2a344e-35ee3012-deb1a827
I0111 20:27:18.762315 140660168441856 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2021-7a5zc"
I0111 20:27:19.193530 140660168441856 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2021-7a5zc"
I0111 20:27:19.590487 140660168441856 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2021-7a5zc"
I0111 20:27:20.057117 140660168441856 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004839673-60eb15fcf1df8-4f1e7131-998b4452
I0111 20:27:24.541139 140660168441856 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:27:34.734542 140660168441856 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:27:34.734833 140660168441856 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2021-7a5zc/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:27:35.280054 140660168441856 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:27:35.407298 140660168441856 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:27:35.407573 140660168441856 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2021-7a5zc
I0111 20:28:05.610698 140660168441856 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2021-7a5zc deleted
I0111 20:28:05.611009 140660168441856 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:28:15.930924 140660168441856 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:28:15.931241 140660168441856 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:28:16.067694 140660168441856 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:28:16.067929 140660168441856 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2021-7a5zc/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:28:16.714301 140660168441856 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:28:16.845678 140660168441856 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:28:16.845938 140660168441856 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2021-7a5zc
I0111 20:29:18.149069 140660168441856 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2021-7a5zc deleted
I0111 20:29:18.149410 140660168441856 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:29:18.149624 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-vv7smh7f:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2021-7a5zc%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-vv7smh7f%22;timeRange=2024-01-11T20:24:13.027167Z%2F2024-01-11T20:28:05.610976Z?project=grpc-testing
I0111 20:29:18.149745 140660168441856 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-13xkr660:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2021-7a5zc%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-13xkr660%22;timeRange=2024-01-11T20:22:42.813943Z%2F2024-01-11T20:29:18.149251Z?project=grpc-testing
[ OK ] OutlierDetectionTest.test_outlier_detection
I0111 20:29:18.149974 140660168441856 base_testcase.py:52] ----- TestCase __main__.OutlierDetectionTest.test_outlier_detection PASSED -----
----------------------------------------------------------------------
Ran 1 test in 488.784s
OK
+ for test in "${test_suites[@]}"
+ run_test remove_neg_test
+ local test_name=remove_neg_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test'
+ set -x
+ python -m tests.remove_neg_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:29:19.523287 140107089178624 xds_k8s_testcase.py:157] ----- Testing RemoveNegTest -----
I0111 20:29:19.523460 140107089178624 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:29:19.523785 140107089178624 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:29:19.540632 140107089178624 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:29:19.558647 140107089178624 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0111 20:29:19.563686 140107089178624 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2029-ibypu
I0111 20:29:19.857897 140107089178624 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/remove_neg_test
I0111 20:29:20.315825 140107089178624 xds_k8s_testcase.py:663] Found unused xds port: 18730
I0111 20:29:20.318073 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0111 20:29:20.318210 140107089178624 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2029-ibypu"
I0111 20:29:20.321232 140107089178624 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2029-ibypu
type: GRPC
...
I0111 20:29:20.974682 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004960381-60eb16700fb0b-ae6a0324-1a79d5cb
I0111 20:29:23.172715 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0111 20:29:23.172899 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0111 20:29:23.172973 140107089178624 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2029-ibypu"
I0111 20:29:23.181327 140107089178624 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2029-ibypu
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2029-ibypu
protocol: GRPC
...
I0111 20:29:23.946090 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004963183-60eb1672bbb22-22d40deb-676e2a27
I0111 20:29:41.212660 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0111 20:29:41.212850 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0111 20:29:41.212925 140107089178624 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2029-ibypu": psm-grpc-server:18730 -> psm-interop-backend-service-20240111-2029-ibypu
I0111 20:29:41.224841 140107089178624 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2029-ibypu
hostRules:
- hosts:
- psm-grpc-server:18730
pathMatcher: psm-interop-path-matcher-20240111-2029-ibypu
name: psm-interop-url-map-20240111-2029-ibypu
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2029-ibypu
name: psm-interop-path-matcher-20240111-2029-ibypu
...
I0111 20:29:41.876422 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004981226-60eb1683f0d53-0ab8d988-f946355a
I0111 20:29:44.069377 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0111 20:29:44.069590 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0111 20:29:44.069682 140107089178624 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2029-ibypu proxy "GRPC" to URL map psm-interop-url-map-20240111-2029-ibypu
I0111 20:29:44.071632 140107089178624 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2029-ibypu
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2029-ibypu
validate_for_proxyless: true
...
I0111 20:29:44.545775 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004984074-60eb1686a7f62-c0d6201b-a92cb6b2
I0111 20:29:46.914426 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0111 20:29:46.914657 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0111 20:29:46.914758 140107089178624 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2029-ibypu" in network "default-vpc": 0.0.0.0:18730 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2029-ibypu
I0111 20:29:46.917364 140107089178624 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2029-ibypu
network: global/networks/default-vpc
portRange: 18730
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2029-ibypu
...
I0111 20:29:47.774327 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705004986919-60eb16895e9d1-9af049f3-e9cd2977
I0111 20:30:16.244135 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0111 20:30:16.244380 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0111 20:30:16.244487 140107089178624 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2029-ibypu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:30:16.244748 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22;timeRange=2024-01-11T20:30:16.244584Z%2F2024-01-11T21:00:16.244584Z?project=grpc-testing
I0111 20:30:16.249692 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2029-ibypu
labels:
name: psm-interop-server-20240111-2029-ibypu
owner: xds-k8s-interop-test
...
I0111 20:30:16.385384 140107089178624 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2029-ibypu created
I0111 20:30:16.388290 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2029-ibypu-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:30:16.459930 140107089178624 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:30:16.460145 140107089178624 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2029-ibypu
I0111 20:30:16.555035 140107089178624 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2029-ibypu: neg_name=psm-interop-server-20240111-2029-ibypu-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:30:16.555249 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:30:17.304565 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:30:17.363783 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:30:17.373074 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2029-ibypu
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-jydquvhe
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2029-ibypu-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:30:17.449811 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:30:17.450053 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:30:17.556433 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-66d9df9454-4tw54']
I0111 20:30:17.556758 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-66d9df9454-4tw54 to start
I0111 20:30:20.802196 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-server-66d9df9454-4tw54 ready, IP: 10.120.3.107
I0111 20:30:20.802464 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-66d9df9454-4tw54 to remove_neg_test/test_app_logs/psm-interop-server-20240111-2029-ibypu_psm-grpc-server-66d9df9454-4tw54.log
I0111 20:30:20.802940 140106995852864 k8s_log_collector.py:70] Starting log collection thread 140106995852864 for psm-grpc-server-66d9df9454-4tw54
I0111 20:30:20.803129 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:30:27.226154 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:30:27.226575 140107089178624 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20240111-2029-ibypu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:30:27.226805 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22;timeRange=2024-01-11T20:30:27.226676Z%2F2024-01-11T21:00:27.226676Z?project=grpc-testing
I0111 20:30:27.226887 140107089178624 k8s_base_runner.py:299] Reusing namespace: psm-interop-server-20240111-2029-ibypu
I0111 20:30:27.281032 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:30:27.328138 140107089178624 k8s_base_runner.py:284] Service psm-grpc-server-alt created
I0111 20:30:27.328392 140107089178624 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server-alt in namespace psm-interop-server-20240111-2029-ibypu
I0111 20:30:27.422993 140107089178624 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server-alt, namespace psm-interop-server-20240111-2029-ibypu: neg_name=psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt, port=8080, zones=['us-central1-a']
I0111 20:30:27.423218 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:30:28.115475 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:30:28.155731 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-server-alt created
I0111 20:30:28.164073 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20240111-2029-ibypu
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-fg9x9et7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:30:28.217071 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-server-alt created
I0111 20:30:28.217321 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0111 20:30:28.300264 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-796b4677-fhv6m']
I0111 20:30:28.300539 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-alt-796b4677-fhv6m to start
I0111 20:30:31.604361 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-server-alt-796b4677-fhv6m ready, IP: 10.120.2.177
I0111 20:30:31.604636 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-alt-796b4677-fhv6m to remove_neg_test/test_app_logs/psm-interop-server-20240111-2029-ibypu_psm-grpc-server-alt-796b4677-fhv6m.log
I0111 20:30:31.605150 140106987460160 k8s_log_collector.py:70] Starting log collection thread 140106987460160 for psm-grpc-server-alt-796b4677-fhv6m
I0111 20:30:31.605299 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0111 20:30:39.053493 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-server-alt has 1 replicas available
I0111 20:30:39.053934 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0111 20:30:39.054048 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0111 20:30:39.090837 140107089178624 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:30:39.282478 140107089178624 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server" in zone us-central1-a
I0111 20:30:39.282725 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:30:39.287132 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:30:40.341223 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005039289-60eb16bb50369-390b13e2-633f668b
I0111 20:31:21.174078 140107089178624 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2029-ibypu to report all backends healthy: ['psm-interop-server-20240111-2029-ibypu-psm-grpc-server']
I0111 20:31:29.673355 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:31:33.757882 140107089178624 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:31:33.889382 140107089178624 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt" in zone us-central1-a
I0111 20:31:33.889635 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:31:33.894786 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:31:34.809871 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005093897-60eb16ef64458-622e0d5a-51aacb32
I0111 20:32:19.804207 140107089178624 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2029-ibypu to report all backends healthy: ['psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt', 'psm-interop-server-20240111-2029-ibypu-psm-grpc-server']
I0111 20:32:20.168598 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:32:24.329395 140107089178624 compute.py:494] Backend psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt in zone us-central1-a reported healthy
I0111 20:32:28.334081 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0111 20:32:28.334320 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0111 20:32:28.334432 140107089178624 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2029-ibypu: server_target=xds:///psm-grpc-server:18730 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:32:28.334674 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2029-ibypu%22;timeRange=2024-01-11T20:32:28.334511Z%2F2024-01-11T21:02:28.334511Z?project=grpc-testing
I0111 20:32:28.336619 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2029-ibypu
labels:
name: psm-interop-client-20240111-2029-ibypu
owner: xds-k8s-interop-test
...
I0111 20:32:28.399598 140107089178624 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2029-ibypu created
I0111 20:32:28.399821 140107089178624 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2029-ibypu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:32:29.090326 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2029-ibypu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:32:29.156451 140107089178624 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:32:29.165004 140107089178624 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2029-ibypu
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-h74zgxc4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:18730"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:32:29.240021 140107089178624 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:32:29.240262 140107089178624 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:32:29.343608 140107089178624 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-85c5579db6-mmtfr']
I0111 20:32:29.343871 140107089178624 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-85c5579db6-mmtfr to start
I0111 20:32:31.534774 140107089178624 k8s_base_runner.py:820] Pod psm-grpc-client-85c5579db6-mmtfr ready, IP: 10.120.0.75
I0111 20:32:31.535062 140107089178624 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-85c5579db6-mmtfr to remove_neg_test/test_app_logs/psm-interop-client-20240111-2029-ibypu_psm-grpc-client-85c5579db6-mmtfr.log
I0111 20:32:31.535550 140106979067456 k8s_log_collector.py:70] Starting log collection thread 140106979067456 for psm-grpc-client-85c5579db6-mmtfr
I0111 20:32:31.535698 140107089178624 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:32:33.718744 140107089178624 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:32:33.719341 140107089178624 client_app.py:284] [psm-grpc-client-85c5579db6-mmtfr] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:32:33.823825 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:32:43.868789 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:32:53.912834 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:03.957633 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:14.002755 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:30.051846 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:55.110978 140107089178624 client_app.py:312] [psm-grpc-client-85c5579db6-mmtfr] xDS control plane channel:
I0111 20:33:57.145411 140107089178624 client_app.py:323] [psm-grpc-client-85c5579db6-mmtfr] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:33:57.145658 140107089178624 client_app.py:294] [psm-grpc-client-85c5579db6-mmtfr] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:33:57.145902 140107089178624 client_app.py:244] [psm-grpc-client-85c5579db6-mmtfr] Waiting to report a READY channel to xds:///psm-grpc-server:18730
I0111 20:33:57.177912 140107089178624 client_app.py:371] [psm-grpc-client-85c5579db6-mmtfr] Server channel:
I0111 20:33:57.210292 140107089178624 client_app.py:384] [psm-grpc-client-85c5579db6-mmtfr] Found subchannel in state READY:
I0111 20:33:57.210682 140107089178624 client_app.py:255] [psm-grpc-client-85c5579db6-mmtfr] Channel to xds:///psm-grpc-server:18730 transitioned to state READY:
I0111 20:33:57.210834 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0111 20:33:57.210911 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0111 20:33:57.211120 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:33:57.245143 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.08_test_client_xds_config_exists ---
I0111 20:33:57.245348 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0111 20:33:57.245553 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:34:01.252782 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-796b4677-fhv6m: 88
psm-grpc-server-66d9df9454-4tw54: 12
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-796b4677-fhv6m: 88
psm-grpc-server-66d9df9454-4tw54: 12
I0111 20:34:01.253057 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.09_test_server_received_rpcs_from_test_client ---
I0111 20:34:01.253152 140107089178624 xds_k8s_testcase.py:269] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0111 20:34:01.253484 140107089178624 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-66d9df9454-4tw54', 'psm-grpc-server-alt-796b4677-fhv6m']
I0111 20:34:01.253638 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:34:05.252764 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-alt-796b4677-fhv6m: 48
psm-grpc-server-66d9df9454-4tw54: 52
rpcs_by_method:
UnaryCall:
psm-grpc-server-alt-796b4677-fhv6m: 48
psm-grpc-server-66d9df9454-4tw54: 52
I0111 20:34:05.403217 140107089178624 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:34:05.703940 140107089178624 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2029-ibypu-psm-grpc-server-alt" in zone us-central1-a
I0111 20:34:05.704170 140107089178624 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2029-ibypu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2029-ibypu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server', zone='us-central1-a')}
I0111 20:34:05.708787 140107089178624 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2029-ibypu-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:34:06.595280 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005245710-60eb17802c23e-fb9a4aad-eb01346e
I0111 20:34:56.023252 140107089178624 xds_k8s_testcase.py:472] Verifying RPCs go to servers ['psm-grpc-server-66d9df9454-4tw54']
I0111 20:34:56.023592 140107089178624 grpc.py:75] [psm-grpc-client-85c5579db6-mmtfr:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:35:00.052741 140107089178624 xds_k8s_testcase.py:601] [psm-grpc-client-85c5579db6-mmtfr] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-66d9df9454-4tw54: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-66d9df9454-4tw54: 100
I0111 20:35:00.053199 140107089178624 xds_k8s_testcase.py:274] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.10_remove_neg ---
I0111 20:35:00.053337 140107089178624 xds_k8s_testcase.py:687] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0111 20:35:00.191177 140107089178624 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2029-ibypu"
I0111 20:35:01.072874 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005300194-60eb17b421d70-d25d64e6-ad7e3c2c
I0111 20:35:29.158085 140107089178624 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2029-ibypu"
I0111 20:35:29.268464 140107089178624 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2029-ibypu"
I0111 20:35:29.691504 140107089178624 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2029-ibypu"
I0111 20:35:30.153193 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005329694-60eb17d043e8c-6b87a1d9-8c4083ad
I0111 20:35:32.544122 140107089178624 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2029-ibypu"
I0111 20:35:33.008916 140107089178624 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2029-ibypu"
I0111 20:35:33.734484 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005333014-60eb17d36e681-938d7eca-7da06934
I0111 20:35:36.074776 140107089178624 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2029-ibypu"
I0111 20:35:36.518625 140107089178624 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2029-ibypu"
I0111 20:35:37.278059 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005336524-60eb17d6c753b-00840ed1-5d890841
I0111 20:36:03.198721 140107089178624 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2029-ibypu"
I0111 20:36:03.714750 140107089178624 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2029-ibypu"
I0111 20:36:04.186576 140107089178624 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2029-ibypu"
I0111 20:36:04.817249 140107089178624 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005364282-60eb17f140499-5f2cd608-bd93e639
I0111 20:36:09.306283 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:36:19.513892 140107089178624 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:36:19.514219 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2029-ibypu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:36:20.077144 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:36:20.194626 140107089178624 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:36:20.194863 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2029-ibypu
I0111 20:36:50.378154 140107089178624 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2029-ibypu deleted
I0111 20:36:50.378445 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:37:00.571100 140107089178624 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:37:00.571406 140107089178624 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:37:00.689821 140107089178624 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:37:00.690067 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:37:01.520982 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:37:01.639441 140107089178624 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:37:01.639869 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2029-ibypu
I0111 20:38:01.886046 140107089178624 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2029-ibypu deleted
I0111 20:38:01.886358 140107089178624 k8s_base_runner.py:708] Deleting deployment psm-grpc-server-alt
I0111 20:38:01.944930 140107089178624 k8s_base_runner.py:726] Deleting service psm-grpc-server-alt
I0111 20:38:01.997323 140107089178624 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2029-ibypu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:38:02.567894 140107089178624 k8s_base_runner.py:742] Deleting service account psm-grpc-server-alt
I0111 20:38:02.624440 140107089178624 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2029-ibypu
I0111 20:38:02.660602 140107089178624 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:38:02.660977 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-h74zgxc4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2029-ibypu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-h74zgxc4%22;timeRange=2024-01-11T20:32:28.334503Z%2F2024-01-11T20:36:50.378410Z?project=grpc-testing
I0111 20:38:02.661146 140107089178624 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-jydquvhe:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2029-ibypu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-jydquvhe%22;timeRange=2024-01-11T20:30:16.244575Z%2F2024-01-11T20:38:01.886324Z?project=grpc-testing
[ OK ] RemoveNegTest.test_remove_neg
I0111 20:38:02.661428 140107089178624 base_testcase.py:52] ----- TestCase __main__.RemoveNegTest.test_remove_neg PASSED -----
----------------------------------------------------------------------
Ran 1 test in 523.141s
OK
+ for test in "${test_suites[@]}"
+ run_test round_robin_test
+ local test_name=round_robin_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test'
+ set -x
+ python -m tests.round_robin_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:38:04.018293 140220474249216 xds_k8s_testcase.py:157] ----- Testing RoundRobinTest -----
I0111 20:38:04.018457 140220474249216 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:38:04.018697 140220474249216 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:38:04.035683 140220474249216 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:38:04.052942 140220474249216 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0111 20:38:04.057728 140220474249216 xds_k8s_testcase.py:668] Test run resource prefix: psm-interop, suffix: 20240111-2038-o77hx
I0111 20:38:04.355690 140220474249216 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/round_robin_test
I0111 20:38:04.728399 140220474249216 xds_k8s_testcase.py:663] Found unused xds port: 5405
I0111 20:38:04.728780 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0111 20:38:04.728887 140220474249216 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2038-o77hx"
I0111 20:38:04.732052 140220474249216 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20240111-2038-o77hx
type: GRPC
...
I0111 20:38:05.473416 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005484797-60eb18642edc1-c9bfd312-264182ec
I0111 20:38:07.856784 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0111 20:38:07.857042 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0111 20:38:07.857143 140220474249216 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2038-o77hx"
I0111 20:38:07.865239 140220474249216 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2038-o77hx
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2038-o77hx
protocol: GRPC
...
I0111 20:38:08.674311 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005487867-60eb18671c564-9daf6266-1b239c55
I0111 20:38:23.742521 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0111 20:38:23.742748 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0111 20:38:23.742840 140220474249216 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2038-o77hx": psm-grpc-server:5405 -> psm-interop-backend-service-20240111-2038-o77hx
I0111 20:38:23.754699 140220474249216 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2038-o77hx
hostRules:
- hosts:
- psm-grpc-server:5405
pathMatcher: psm-interop-path-matcher-20240111-2038-o77hx
name: psm-interop-url-map-20240111-2038-o77hx
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2038-o77hx
name: psm-interop-path-matcher-20240111-2038-o77hx
...
I0111 20:38:24.422335 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005503757-60eb187643b87-c5ba0228-ec91a1a8
I0111 20:38:26.717602 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0111 20:38:26.717848 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0111 20:38:26.717947 140220474249216 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2038-o77hx proxy "GRPC" to URL map psm-interop-url-map-20240111-2038-o77hx
I0111 20:38:26.720094 140220474249216 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2038-o77hx
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2038-o77hx
validate_for_proxyless: true
...
I0111 20:38:27.276390 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005506722-60eb187917988-997898ee-2cd5b27e
I0111 20:38:29.666043 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0111 20:38:29.666253 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0111 20:38:29.666373 140220474249216 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2038-o77hx" in network "default-vpc": 0.0.0.0:5405 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2038-o77hx
I0111 20:38:29.668943 140220474249216 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2038-o77hx
network: global/networks/default-vpc
portRange: 5405
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2038-o77hx
...
I0111 20:38:30.442459 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005509673-60eb187be82ea-89b0d0e7-ea5f4315
I0111 20:38:51.959462 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0111 20:38:51.959719 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0111 20:38:51.959822 140220474249216 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2038-o77hx: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0111 20:38:51.960082 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2038-o77hx%22;timeRange=2024-01-11T20:38:51.959923Z%2F2024-01-11T21:08:51.959923Z?project=grpc-testing
I0111 20:38:51.964933 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2038-o77hx
labels:
name: psm-interop-server-20240111-2038-o77hx
owner: xds-k8s-interop-test
...
I0111 20:38:52.127088 140220474249216 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2038-o77hx created
I0111 20:38:52.129953 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2038-o77hx-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:38:52.202089 140220474249216 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:38:52.202331 140220474249216 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2038-o77hx
I0111 20:38:52.303960 140220474249216 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2038-o77hx: neg_name=psm-interop-server-20240111-2038-o77hx-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:38:52.304182 140220474249216 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2038-o77hx/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:38:53.019284 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:38:53.079222 140220474249216 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:38:53.087938 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2038-o77hx
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nbgcxtq1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2038-o77hx-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:38:53.166104 140220474249216 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:38:53.166345 140220474249216 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0111 20:38:53.322724 140220474249216 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-6cfcf655dd-lsbbq', 'psm-grpc-server-6cfcf655dd-p2nf6']
I0111 20:38:53.323058 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cfcf655dd-lsbbq to start
I0111 20:38:56.556079 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-server-6cfcf655dd-lsbbq ready, IP: 10.120.2.179
I0111 20:38:56.556338 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cfcf655dd-lsbbq to round_robin_test/test_app_logs/psm-interop-server-20240111-2038-o77hx_psm-grpc-server-6cfcf655dd-lsbbq.log
I0111 20:38:56.556788 140220380472896 k8s_log_collector.py:70] Starting log collection thread 140220380472896 for psm-grpc-server-6cfcf655dd-lsbbq
I0111 20:38:56.556936 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cfcf655dd-p2nf6 to start
I0111 20:38:56.730018 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-server-6cfcf655dd-p2nf6 ready, IP: 10.120.0.76
I0111 20:38:56.730258 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cfcf655dd-p2nf6 to round_robin_test/test_app_logs/psm-interop-server-20240111-2038-o77hx_psm-grpc-server-6cfcf655dd-p2nf6.log
I0111 20:38:56.730705 140220372080192 k8s_log_collector.py:70] Starting log collection thread 140220372080192 for psm-grpc-server-6cfcf655dd-p2nf6
I0111 20:38:56.730841 140220474249216 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0111 20:39:03.219086 140220474249216 k8s_base_runner.py:788] Deployment psm-grpc-server has 2 replicas available
I0111 20:39:03.219639 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0111 20:39:03.219774 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0111 20:39:03.256953 140220474249216 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:39:03.597664 140220474249216 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2038-o77hx-psm-grpc-server" in zone us-central1-a
I0111 20:39:03.597927 140220474249216 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2038-o77hx: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2038-o77hx-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2038-o77hx-psm-grpc-server', zone='us-central1-a')}
I0111 20:39:03.602385 140220474249216 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2038-o77hx-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:39:04.437691 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005543604-60eb189c44086-0c162087-12b3a6bf
I0111 20:39:45.558553 140220474249216 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2038-o77hx to report all backends healthy: ['psm-interop-server-20240111-2038-o77hx-psm-grpc-server']
I0111 20:39:58.389801 140220474249216 compute.py:494] Backend psm-interop-server-20240111-2038-o77hx-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:40:02.394601 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0111 20:40:02.394835 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0111 20:40:02.394946 140220474249216 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2038-o77hx: server_target=xds:///psm-grpc-server:5405 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:40:02.395180 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2038-o77hx%22;timeRange=2024-01-11T20:40:02.395048Z%2F2024-01-11T21:10:02.395048Z?project=grpc-testing
I0111 20:40:02.397136 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2038-o77hx
labels:
name: psm-interop-client-20240111-2038-o77hx
owner: xds-k8s-interop-test
...
I0111 20:40:02.439748 140220474249216 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2038-o77hx created
I0111 20:40:02.440038 140220474249216 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2038-o77hx/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:40:03.166345 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2038-o77hx
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:40:03.230656 140220474249216 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:40:03.239415 140220474249216 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2038-o77hx
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-uh9k1av0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:5405"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:40:03.315493 140220474249216 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:40:03.315780 140220474249216 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:40:03.423839 140220474249216 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5b6d5cdb69-k5n92']
I0111 20:40:03.424149 140220474249216 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-5b6d5cdb69-k5n92 to start
I0111 20:40:07.704100 140220474249216 k8s_base_runner.py:820] Pod psm-grpc-client-5b6d5cdb69-k5n92 ready, IP: 10.120.3.109
I0111 20:40:07.704365 140220474249216 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-5b6d5cdb69-k5n92 to round_robin_test/test_app_logs/psm-interop-client-20240111-2038-o77hx_psm-grpc-client-5b6d5cdb69-k5n92.log
I0111 20:40:07.704831 140220363687488 k8s_log_collector.py:70] Starting log collection thread 140220363687488 for psm-grpc-client-5b6d5cdb69-k5n92
I0111 20:40:07.704979 140220474249216 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:40:09.883449 140220474249216 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:40:09.884044 140220474249216 client_app.py:284] [psm-grpc-client-5b6d5cdb69-k5n92] ADS: Waiting for active calls to xDS control plane to trafficdirector.googleapis.com:443
I0111 20:40:09.986282 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:20.031319 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:30.076177 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:40.120726 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:40:50.165464 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:06.214444 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:31.274459 140220474249216 client_app.py:312] [psm-grpc-client-5b6d5cdb69-k5n92] xDS control plane channel:
I0111 20:41:33.309262 140220474249216 client_app.py:323] [psm-grpc-client-5b6d5cdb69-k5n92] Detected active calls to xDS control plane trafficdirector.googleapis.com:443, channel:
I0111 20:41:33.309552 140220474249216 client_app.py:294] [psm-grpc-client-5b6d5cdb69-k5n92] ADS: Detected active calls to xDS control plane trafficdirector.googleapis.com:443
I0111 20:41:33.309814 140220474249216 client_app.py:244] [psm-grpc-client-5b6d5cdb69-k5n92] Waiting to report a READY channel to xds:///psm-grpc-server:5405
I0111 20:41:33.342113 140220474249216 client_app.py:371] [psm-grpc-client-5b6d5cdb69-k5n92] Server channel:
I0111 20:41:33.374472 140220474249216 client_app.py:384] [psm-grpc-client-5b6d5cdb69-k5n92] Found subchannel in state READY:
I0111 20:41:33.374861 140220474249216 client_app.py:255] [psm-grpc-client-5b6d5cdb69-k5n92] Channel to xds:///psm-grpc-server:5405 transitioned to state READY:
I0111 20:41:33.375025 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0111 20:41:33.375128 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0111 20:41:33.375325 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:41:33.409441 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.08_test_client_xds_config_exists ---
I0111 20:41:33.409652 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0111 20:41:33.409900 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:41:37.407031 140220474249216 xds_k8s_testcase.py:601] [psm-grpc-client-5b6d5cdb69-k5n92] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
I0111 20:41:37.407300 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.09_test_server_received_rpcs_from_test_client ---
I0111 20:41:37.407393 140220474249216 xds_k8s_testcase.py:269] --- Starting subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0111 20:41:37.407519 140220474249216 grpc.py:75] [psm-grpc-client-5b6d5cdb69-k5n92:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:41:41.406090 140220474249216 xds_k8s_testcase.py:601] [psm-grpc-client-5b6d5cdb69-k5n92] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cfcf655dd-p2nf6: 50
psm-grpc-server-6cfcf655dd-lsbbq: 50
I0111 20:41:41.406372 140220474249216 xds_k8s_testcase.py:274] --- Finished subTest __main__.RoundRobinTest.test_round_robin.10_round_robin ---
I0111 20:41:41.406490 140220474249216 xds_k8s_testcase.py:687] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0111 20:41:41.619422 140220474249216 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2038-o77hx"
I0111 20:41:42.323336 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005701623-60eb1932f6df6-0cafd3d9-af03f35c
I0111 20:42:10.679920 140220474249216 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2038-o77hx"
I0111 20:42:10.791017 140220474249216 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2038-o77hx"
I0111 20:42:11.199696 140220474249216 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2038-o77hx"
I0111 20:42:11.797508 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005731202-60eb194f2c689-053f8795-55058474
I0111 20:42:14.096377 140220474249216 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2038-o77hx"
I0111 20:42:14.505065 140220474249216 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2038-o77hx"
I0111 20:42:15.345269 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005734510-60eb195253fb6-c287a5a9-4c6203aa
I0111 20:42:17.700789 140220474249216 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2038-o77hx"
I0111 20:42:18.068951 140220474249216 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2038-o77hx"
I0111 20:42:18.782963 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005738074-60eb1955ba023-084515bc-ff76eb75
I0111 20:42:44.716268 140220474249216 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2038-o77hx"
I0111 20:42:45.213931 140220474249216 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2038-o77hx"
I0111 20:42:45.688411 140220474249216 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2038-o77hx"
I0111 20:42:46.205750 140220474249216 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005765783-60eb197026f1a-1d348629-a0ac992c
I0111 20:42:50.756645 140220474249216 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:43:00.933695 140220474249216 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:43:00.934101 140220474249216 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2038-o77hx/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:43:01.533618 140220474249216 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:43:01.656221 140220474249216 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:43:01.656483 140220474249216 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2038-o77hx
I0111 20:43:31.842261 140220474249216 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2038-o77hx deleted
I0111 20:43:31.842575 140220474249216 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:43:42.058790 140220474249216 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:43:42.059175 140220474249216 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:43:42.181828 140220474249216 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:43:42.182139 140220474249216 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2038-o77hx/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:43:42.782669 140220474249216 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:43:42.898220 140220474249216 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:43:42.898451 140220474249216 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2038-o77hx
I0111 20:44:13.078806 140220474249216 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2038-o77hx deleted
I0111 20:44:13.079144 140220474249216 xds_k8s_testcase.py:711] ----- Test client/server logs -----
I0111 20:44:13.079380 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client-uh9k1av0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2038-o77hx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-uh9k1av0%22;timeRange=2024-01-11T20:40:02.395040Z%2F2024-01-11T20:43:31.842538Z?project=grpc-testing
I0111 20:44:13.079509 140220474249216 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server-nbgcxtq1:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2038-o77hx%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-nbgcxtq1%22;timeRange=2024-01-11T20:38:51.959915Z%2F2024-01-11T20:44:13.078990Z?project=grpc-testing
[ OK ] RoundRobinTest.test_round_robin
I0111 20:44:13.079764 140220474249216 base_testcase.py:52] ----- TestCase __main__.RoundRobinTest.test_round_robin PASSED -----
----------------------------------------------------------------------
Ran 1 test in 369.064s
OK
+ for test in "${test_suites[@]}"
+ run_test bootstrap_generator_test
+ local test_name=bootstrap_generator_test
+ local out_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test'
+ set -x
+ python -m tests.bootstrap_generator_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --server_image=gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b --client_image=gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b --testing_version=dev-master --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test --xml_output_file=/tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test/sponge_log.log
Running tests under Python 3.10.12: /tmp/tmp.mnM8lQRITP/psm-interop/venv/bin/python
I0111 20:44:14.519476 140142540660736 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorClientTest -----
I0111 20:44:14.519663 140142540660736 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:44:14.519982 140142540660736 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:44:14.536901 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:44:14.553819 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0111 20:44:14.558431 140142540660736 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240111-2044-sv6uq
I0111 20:44:15.118297 140142540660736 bootstrap_generator_testcase.py:88] Found unused xds port: 26747
I0111 20:44:15.118662 140142540660736 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2044-sv6uq"
I0111 20:44:15.122098 140142540660736 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240111-2044-sv6uq
type: GRPC
...
I0111 20:44:15.754775 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005855182-60eb19c568f98-e50fc3f2-742257b1
I0111 20:44:18.207105 140142540660736 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2044-sv6uq"
I0111 20:44:18.215328 140142540660736 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2044-sv6uq
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2044-sv6uq
protocol: GRPC
...
I0111 20:44:19.089204 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005858218-60eb19c84e14f-481253f9-93b9b277
I0111 20:44:34.387290 140142540660736 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2044-sv6uq": psm-grpc-server:26747 -> psm-interop-backend-service-20240111-2044-sv6uq
I0111 20:44:34.398491 140142540660736 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2044-sv6uq
hostRules:
- hosts:
- psm-grpc-server:26747
pathMatcher: psm-interop-path-matcher-20240111-2044-sv6uq
name: psm-interop-url-map-20240111-2044-sv6uq
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2044-sv6uq
name: psm-interop-path-matcher-20240111-2044-sv6uq
...
I0111 20:44:35.099933 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005874400-60eb19d7bcd41-53aa7a7b-f99e4c44
I0111 20:44:37.449626 140142540660736 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2044-sv6uq proxy "GRPC" to URL map psm-interop-url-map-20240111-2044-sv6uq
I0111 20:44:37.451752 140142540660736 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2044-sv6uq
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2044-sv6uq
validate_for_proxyless: true
...
I0111 20:44:37.942819 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005877454-60eb19daa64e3-bc29cdab-0b570a0f
I0111 20:44:40.226653 140142540660736 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2044-sv6uq" in network "default-vpc": 0.0.0.0:26747 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2044-sv6uq
I0111 20:44:40.229472 140142540660736 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2044-sv6uq
network: global/networks/default-vpc
portRange: 26747
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2044-sv6uq
...
I0111 20:44:41.151144 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005880231-60eb19dd4c5d4-1c563b7c-761c367e
I0111 20:44:54.207493 140142540660736 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/go/master/branch/xds_k8s_lb/bootstrap_generator_test
I0111 20:44:54.329736 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2044-sv6uq: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:44:54.330115 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:44:54.329955Z%2F2024-01-11T21:14:54.329955Z?project=grpc-testing
I0111 20:44:54.335117 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2044-sv6uq
labels:
name: psm-interop-server-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:44:54.465403 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2044-sv6uq created
I0111 20:44:54.468483 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2044-sv6uq-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:44:54.541352 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:44:54.541559 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2044-sv6uq
I0111 20:44:54.641555 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2044-sv6uq: neg_name=psm-interop-server-20240111-2044-sv6uq-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:44:54.641776 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2044-sv6uq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:44:55.319915 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:44:55.381590 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:44:55.391147 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2044-sv6uq
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-w7el8zc1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2044-sv6uq-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:44:55.457427 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:44:55.457651 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:44:55.560288 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6454f4b8bd-7f6bb']
I0111 20:44:55.560539 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6454f4b8bd-7f6bb to start
I0111 20:44:58.800908 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-6454f4b8bd-7f6bb ready, IP: 10.120.2.180
I0111 20:44:58.801188 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6454f4b8bd-7f6bb to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2044-sv6uq_psm-grpc-server-6454f4b8bd-7f6bb.log
I0111 20:44:58.801652 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-6454f4b8bd-7f6bb
I0111 20:44:58.801825 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:45:05.221496 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:45:05.258428 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:45:05.395655 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2044-sv6uq-psm-grpc-server" in zone us-central1-a
I0111 20:45:05.395876 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2044-sv6uq: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2044-sv6uq-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2044-sv6uq-psm-grpc-server', zone='us-central1-a')}
I0111 20:45:05.400449 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2044-sv6uq-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:45:06.353496 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705005905402-60eb19f54db7f-f2326072-bf036fdd
I0111 20:45:57.965567 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2044-sv6uq to report all backends healthy: ['psm-interop-server-20240111-2044-sv6uq-psm-grpc-server']
I0111 20:46:11.082528 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2044-sv6uq-psm-grpc-server in zone us-central1-a reported healthy
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:46:15.087495 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.14.0 -----
I0111 20:46:15.089928 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:46:15.090277 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:46:15.090115Z%2F2024-01-11T21:16:15.090115Z?project=grpc-testing
I0111 20:46:15.092195 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:46:15.246905 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:46:15.247134 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:46:15.890919 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:46:15.946979 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:46:15.955327 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ssnn81rl
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:46:16.029316 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:46:16.029568 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:46:16.140331 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-64bcdb54fd-f92ls']
I0111 20:46:16.140623 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-64bcdb54fd-f92ls to start
I0111 20:46:18.332383 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-64bcdb54fd-f92ls ready, IP: 10.120.0.77
I0111 20:46:18.332676 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-64bcdb54fd-f92ls to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-64bcdb54fd-f92ls.log
I0111 20:46:18.333166 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-64bcdb54fd-f92ls
I0111 20:46:18.333339 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:46:20.582031 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:46:20.582584 140142540660736 client_app.py:244] [psm-grpc-client-64bcdb54fd-f92ls] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:46:20.686862 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:30.731376 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:40.776571 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:46:50.821987 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:00.867419 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:16.916966 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:47:41.977340 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:48:07.037137 140142540660736 client_app.py:371] [psm-grpc-client-64bcdb54fd-f92ls] Server channel:
I0111 20:48:07.069717 140142540660736 client_app.py:384] [psm-grpc-client-64bcdb54fd-f92ls] Found subchannel in state READY:
I0111 20:48:07.069964 140142540660736 client_app.py:255] [psm-grpc-client-64bcdb54fd-f92ls] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:48:07.070204 140142540660736 grpc.py:75] [psm-grpc-client-64bcdb54fd-f92ls:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:48:07.104833 140142540660736 grpc.py:75] [psm-grpc-client-64bcdb54fd-f92ls:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:48:11.121255 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-64bcdb54fd-f92ls] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:48:11.121552 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0111 20:48:11.121896 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:48:21.393244 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:48:21.393556 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:48:22.027455 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:48:22.141269 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:48:22.141526 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:48:52.328513 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:48:52.329218 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 20:48:52.329458 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.13.0 -----
I0111 20:48:52.331351 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:48:52.331620 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:48:52.331473Z%2F2024-01-11T21:18:52.331473Z?project=grpc-testing
I0111 20:48:52.333432 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:48:52.395277 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:48:52.395504 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:48:52.982383 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:48:53.035788 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:48:53.043929 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-a1xrr9s1
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:48:53.118485 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:48:53.118724 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:48:53.224979 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-8586888567-s9xnk']
I0111 20:48:53.225250 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-8586888567-s9xnk to start
I0111 20:48:56.468243 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-8586888567-s9xnk ready, IP: 10.120.0.78
I0111 20:48:56.468545 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-8586888567-s9xnk to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-8586888567-s9xnk.log
I0111 20:48:56.468957 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-8586888567-s9xnk
I0111 20:48:56.469113 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:48:57.588819 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:48:57.589401 140142540660736 client_app.py:244] [psm-grpc-client-8586888567-s9xnk] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:48:57.686754 140142540660736 client_app.py:371] [psm-grpc-client-8586888567-s9xnk] Server channel:
I0111 20:48:57.718690 140142540660736 client_app.py:384] [psm-grpc-client-8586888567-s9xnk] Found subchannel in state READY:
I0111 20:48:57.719112 140142540660736 client_app.py:255] [psm-grpc-client-8586888567-s9xnk] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:48:57.719382 140142540660736 grpc.py:75] [psm-grpc-client-8586888567-s9xnk:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:48:57.752972 140142540660736 grpc.py:75] [psm-grpc-client-8586888567-s9xnk:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:49:01.784570 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-8586888567-s9xnk] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:49:01.784882 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0111 20:49:01.785272 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:49:11.936257 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:49:11.936563 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:49:12.486939 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:49:12.561225 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:49:12.561454 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:49:42.704245 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 20:49:42.704996 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 20:49:42.705246 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.12.0 -----
I0111 20:49:42.707194 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:49:42.707479 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:49:42.707362Z%2F2024-01-11T21:19:42.707362Z?project=grpc-testing
I0111 20:49:42.709454 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:49:42.751939 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:49:42.752166 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:49:43.386118 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:49:43.426364 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:49:43.434303 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-3wa3siq7
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:49:43.488190 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:49:43.488429 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:49:43.575676 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-58fcf9686d-czbd6']
I0111 20:49:43.575960 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-58fcf9686d-czbd6 to start
I0111 20:49:46.815373 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-58fcf9686d-czbd6 ready, IP: 10.120.3.111
I0111 20:49:46.815662 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-58fcf9686d-czbd6 to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-58fcf9686d-czbd6.log
I0111 20:49:46.816020 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-58fcf9686d-czbd6
I0111 20:49:46.816153 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:49:47.936704 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:49:47.937278 140142540660736 client_app.py:244] [psm-grpc-client-58fcf9686d-czbd6] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:49:48.037058 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:49:58.082522 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:08.127722 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:18.173596 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:28.218681 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:50:44.268203 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:51:09.328310 140142540660736 client_app.py:371] [psm-grpc-client-58fcf9686d-czbd6] Server channel:
I0111 20:51:09.361263 140142540660736 client_app.py:384] [psm-grpc-client-58fcf9686d-czbd6] Found subchannel in state READY:
I0111 20:51:09.361513 140142540660736 client_app.py:255] [psm-grpc-client-58fcf9686d-czbd6] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:51:09.361730 140142540660736 grpc.py:75] [psm-grpc-client-58fcf9686d-czbd6:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:51:09.395957 140142540660736 grpc.py:75] [psm-grpc-client-58fcf9686d-czbd6:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:51:13.395061 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-58fcf9686d-czbd6] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:51:13.395328 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0111 20:51:13.395743 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:51:23.577870 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:51:23.578180 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:51:24.195605 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:51:24.308147 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:51:24.308376 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:51:54.489159 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 20:51:54.489754 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 20:51:54.489979 140142540660736 bootstrap_generator_test.py:155] ----- testing bootstrap generator version v0.11.0 -----
I0111 20:51:54.491907 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2044-sv6uq: server_target=xds:///psm-grpc-server:26747 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:51:54.492170 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2044-sv6uq%22;timeRange=2024-01-11T20:51:54.492060Z%2F2024-01-11T21:21:54.492060Z?project=grpc-testing
I0111 20:51:54.493946 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2044-sv6uq
labels:
name: psm-interop-client-20240111-2044-sv6uq
owner: xds-k8s-interop-test
...
I0111 20:51:54.553177 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2044-sv6uq created
I0111 20:51:54.553392 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:51:55.088105 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:51:55.152412 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:51:55.160494 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2044-sv6uq
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-6zz6dbae
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:26747"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:51:55.229956 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:51:55.230192 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:51:55.336387 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7fd84cbb98-xkwlc']
I0111 20:51:55.336656 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7fd84cbb98-xkwlc to start
I0111 20:51:58.579613 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-7fd84cbb98-xkwlc ready, IP: 10.120.3.112
I0111 20:51:58.579880 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7fd84cbb98-xkwlc to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2044-sv6uq_psm-grpc-client-7fd84cbb98-xkwlc.log
I0111 20:51:58.580292 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-7fd84cbb98-xkwlc
I0111 20:51:58.580427 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:51:59.700123 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:51:59.700702 140142540660736 client_app.py:244] [psm-grpc-client-7fd84cbb98-xkwlc] Waiting to report a READY channel to xds:///psm-grpc-server:26747
I0111 20:51:59.801590 140142540660736 client_app.py:371] [psm-grpc-client-7fd84cbb98-xkwlc] Server channel:
I0111 20:51:59.834794 140142540660736 client_app.py:384] [psm-grpc-client-7fd84cbb98-xkwlc] Found subchannel in state READY:
I0111 20:51:59.835221 140142540660736 client_app.py:255] [psm-grpc-client-7fd84cbb98-xkwlc] Channel to xds:///psm-grpc-server:26747 transitioned to state READY:
I0111 20:51:59.835451 140142540660736 grpc.py:75] [psm-grpc-client-7fd84cbb98-xkwlc:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 20:51:59.870186 140142540660736 grpc.py:75] [psm-grpc-client-7fd84cbb98-xkwlc:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 20:52:03.885331 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-7fd84cbb98-xkwlc] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6454f4b8bd-7f6bb: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6454f4b8bd-7f6bb: 100
I0111 20:52:03.885614 140142540660736 bootstrap_generator_test.py:133] ----- TestMethod __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0111 20:52:03.885957 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 20:52:14.018023 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 20:52:14.018325 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2044-sv6uq/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:52:14.549336 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 20:52:14.624279 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 20:52:14.624495 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2044-sv6uq
I0111 20:52:44.766071 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2044-sv6uq deleted
[ OK ] BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 20:52:44.766637 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorClientTest.test_baseline_in_client_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0111 20:52:44.824572 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:52:45.145194 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2044-sv6uq-psm-grpc-server" in zone us-central1-a
I0111 20:52:45.145443 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2044-sv6uq: set()
I0111 20:52:45.149304 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 20:52:45.859028 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006365151-60eb1babc104e-9e3a5209-91286722
I0111 20:53:31.095795 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 20:53:41.287217 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 20:53:41.287512 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 20:53:41.415049 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 20:53:41.415308 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2044-sv6uq/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:53:42.003391 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 20:53:42.121703 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 20:53:42.121989 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2044-sv6uq
I0111 20:54:42.363481 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2044-sv6uq deleted
I0111 20:54:42.363807 140142540660736 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2044-sv6uq"
I0111 20:54:43.019310 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006482368-60eb1c1b8a8da-5f6a418c-4ebc2c37
I0111 20:54:51.783479 140142540660736 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2044-sv6uq"
I0111 20:54:51.869429 140142540660736 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2044-sv6uq"
I0111 20:54:52.257336 140142540660736 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2044-sv6uq"
I0111 20:54:52.744244 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006492260-60eb1c24f97db-16aed3a4-b1edf4f7
I0111 20:54:54.996236 140142540660736 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2044-sv6uq"
I0111 20:54:55.378980 140142540660736 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2044-sv6uq"
I0111 20:54:56.174697 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006495384-60eb1c27f4448-cb0b4d27-c1636ae8
I0111 20:55:00.636589 140142540660736 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2044-sv6uq"
I0111 20:55:01.107977 140142540660736 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2044-sv6uq"
I0111 20:55:01.814455 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006501113-60eb1c2d6ae9d-546214cb-3b3caa96
I0111 20:55:14.896846 140142540660736 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2044-sv6uq"
I0111 20:55:15.296034 140142540660736 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2044-sv6uq"
I0111 20:55:15.674203 140142540660736 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2044-sv6uq"
I0111 20:55:16.213811 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006515749-60eb1c3b60141-e83b6290-6871796f
I0111 20:55:20.710044 140142540660736 xds_k8s_testcase.py:157] ----- Testing BootstrapGeneratorServerTest -----
I0111 20:55:20.710307 140142540660736 xds_k8s_testcase.py:158] Logs timezone: UTC
I0111 20:55:20.711016 140142540660736 xds_k8s_testcase.py:106] Detected language and version: TestConfig(client_lang='go', server_lang='go', version='dev-master')
I0111 20:55:20.727207 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0111 20:55:20.743434 140142540660736 k8s.py:243] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
I0111 20:55:20.744117 140142540660736 bootstrap_generator_testcase.py:57] Test run resource prefix: psm-interop, suffix: 20240111-2055-avpqw
I0111 20:55:21.282765 140142540660736 bootstrap_generator_testcase.py:88] Found unused xds port: 4923
I0111 20:55:21.283052 140142540660736 traffic_director.py:191] Creating GRPC Health Check "psm-interop-health-check-20240111-2055-avpqw"
I0111 20:55:21.286405 140142540660736 compute.py:537] Creating compute resource:
---
grpcHealthCheck:
port: 8080
portSpecification: USE_FIXED_PORT
name: psm-interop-health-check-20240111-2055-avpqw
type: GRPC
...
I0111 20:55:22.129929 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006521336-60eb1c40b421b-a13c5a1e-777df0be
I0111 20:55:24.342421 140142540660736 traffic_director.py:218] Creating GRPC Backend Service "psm-interop-backend-service-20240111-2055-avpqw"
I0111 20:55:24.350029 140142540660736 compute.py:537] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20240111-2055-avpqw
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20240111-2055-avpqw
protocol: GRPC
...
I0111 20:55:25.180847 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006524352-60eb1c43946d4-6472bcdc-4583e16c
I0111 20:55:59.598564 140142540660736 traffic_director.py:479] Creating URL map "psm-interop-url-map-20240111-2055-avpqw": psm-grpc-server:4923 -> psm-interop-backend-service-20240111-2055-avpqw
I0111 20:55:59.609861 140142540660736 compute.py:537] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2055-avpqw
hostRules:
- hosts:
- psm-grpc-server:4923
pathMatcher: psm-interop-path-matcher-20240111-2055-avpqw
name: psm-interop-url-map-20240111-2055-avpqw
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20240111-2055-avpqw
name: psm-interop-path-matcher-20240111-2055-avpqw
...
I0111 20:56:00.343596 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006559611-60eb1c6534c26-9954083c-727b754d
I0111 20:56:02.593755 140142540660736 traffic_director.py:578] Creating target psm-interop-target-proxy-20240111-2055-avpqw proxy "GRPC" to URL map psm-interop-url-map-20240111-2055-avpqw
I0111 20:56:02.595804 140142540660736 compute.py:537] Creating compute resource:
---
name: psm-interop-target-proxy-20240111-2055-avpqw
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20240111-2055-avpqw
validate_for_proxyless: true
...
I0111 20:56:03.276043 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006562597-60eb1c680db4c-c750e6c1-93d67d3f
I0111 20:56:05.669324 140142540660736 traffic_director.py:654] Creating forwarding rule "psm-interop-forwarding-rule-20240111-2055-avpqw" in network "default-vpc": 0.0.0.0:4923 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2055-avpqw
I0111 20:56:05.672167 140142540660736 compute.py:537] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20240111-2055-avpqw
network: global/networks/default-vpc
portRange: 4923
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20240111-2055-avpqw
...
I0111 20:56:06.412389 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006565674-60eb1c6afcc98-01c6ad64-566bc799
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 20:56:34.533740 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.14.0 -----
I0111 20:56:34.656663 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 20:56:34.657051 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T20:56:34.656889Z%2F2024-01-11T21:26:34.656889Z?project=grpc-testing
I0111 20:56:34.658848 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 20:56:34.816548 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 20:56:34.819331 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 20:56:34.893602 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 20:56:34.893861 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 20:56:34.988657 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 20:56:34.988928 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:56:35.675130 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:56:35.741965 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 20:56:35.750231 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-xyo7npue
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:56:35.814664 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 20:56:35.814926 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 20:56:35.918545 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-db7cfdd4f-qm7g8']
I0111 20:56:35.918818 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-db7cfdd4f-qm7g8 to start
I0111 20:56:39.160809 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-db7cfdd4f-qm7g8 ready, IP: 10.120.2.182
I0111 20:56:39.161074 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-db7cfdd4f-qm7g8 to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-db7cfdd4f-qm7g8.log
I0111 20:56:39.161428 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-db7cfdd4f-qm7g8
I0111 20:56:39.161561 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 20:56:45.507456 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 20:56:45.544878 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 20:56:45.844813 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 20:56:45.845058 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 20:56:45.849550 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 20:56:46.680086 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006605851-60eb1c914dd12-671f4010-630a8466
I0111 20:57:31.893793 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 20:57:57.426590 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 20:58:01.433539 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 20:58:01.433985 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T20:58:01.433813Z%2F2024-01-11T21:28:01.433813Z?project=grpc-testing
I0111 20:58:01.436079 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 20:58:01.478648 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 20:58:01.478894 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 20:58:02.146666 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 20:58:02.206452 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 20:58:02.215576 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-fc0z13mm
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 20:58:02.285398 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 20:58:02.285646 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 20:58:02.388480 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-f4695bfc7-f85b2']
I0111 20:58:02.388740 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-f4695bfc7-f85b2 to start
I0111 20:58:05.634322 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-f4695bfc7-f85b2 ready, IP: 10.120.3.115
I0111 20:58:05.634583 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-f4695bfc7-f85b2 to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-f4695bfc7-f85b2.log
I0111 20:58:05.634940 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-f4695bfc7-f85b2
I0111 20:58:05.635087 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 20:58:06.773778 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 20:58:06.774339 140142540660736 client_app.py:244] [psm-grpc-client-f4695bfc7-f85b2] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 20:58:06.874165 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:16.920148 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:26.965694 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:37.010961 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:58:47.055922 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:03.105623 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:28.165620 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 20:59:53.225650 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 21:00:18.285653 140142540660736 client_app.py:371] [psm-grpc-client-f4695bfc7-f85b2] Server channel:
I0111 21:00:18.318724 140142540660736 client_app.py:384] [psm-grpc-client-f4695bfc7-f85b2] Found subchannel in state READY:
I0111 21:00:18.319109 140142540660736 client_app.py:255] [psm-grpc-client-f4695bfc7-f85b2] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:00:18.319360 140142540660736 grpc.py:75] [psm-grpc-client-f4695bfc7-f85b2:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:00:18.353782 140142540660736 grpc.py:75] [psm-grpc-client-f4695bfc7-f85b2:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:00:22.381894 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-f4695bfc7-f85b2] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-db7cfdd4f-qm7g8: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-db7cfdd4f-qm7g8: 100
I0111 21:00:22.382216 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') teardown -----
I0111 21:00:22.382571 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:00:32.664484 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:00:32.664852 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:00:33.316529 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:00:33.442970 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:00:33.443211 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:01:03.635258 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:01:03.701334 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:01:03.811141 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:01:03.811381 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:01:03.815282 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:01:04.777332 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705006863817-60eb1d8751c57-03b79d28-4a1b2c7d
I0111 21:01:45.850589 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:01:56.047597 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:01:56.047880 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:01:56.162749 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:01:56.163047 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:01:56.787397 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:01:56.915694 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:01:56.915993 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:03:28.308688 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599')
I0111 21:03:28.309199 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version0 ('v0.14.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 21:03:28.309388 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.13.0 -----
I0111 21:03:28.311281 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:03:28.311493 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:03:28.311385Z%2F2024-01-11T21:33:28.311385Z?project=grpc-testing
I0111 21:03:28.313409 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:03:28.373173 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:03:28.376740 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:03:28.441478 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:03:28.441701 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:03:28.534601 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:03:28.534817 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:03:29.195653 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:03:29.257096 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:03:29.265048 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n4tgfm2h
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:03:29.331502 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:03:29.331758 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:03:29.434362 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6cc98d756c-df447']
I0111 21:03:29.434633 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-6cc98d756c-df447 to start
I0111 21:03:32.666797 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-6cc98d756c-df447 ready, IP: 10.120.2.184
I0111 21:03:32.667063 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-6cc98d756c-df447 to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-6cc98d756c-df447.log
I0111 21:03:32.667413 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-6cc98d756c-df447
I0111 21:03:32.667565 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:03:40.123493 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:03:40.161025 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:03:40.435153 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:03:40.435380 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:03:40.439661 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:03:41.807591 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007020441-60eb1e1cb0125-958b35b5-2f2c006e
I0111 21:04:40.042250 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:04:40.306714 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:04:44.314125 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:04:44.314459 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:04:44.314342Z%2F2024-01-11T21:34:44.314342Z?project=grpc-testing
I0111 21:04:44.316317 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:04:44.358509 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:04:44.358735 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:04:45.004599 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:04:45.071032 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:04:45.079193 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-xstmfz45
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:04:45.157081 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:04:45.157317 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:04:45.269329 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-67cb58cc7f-v69dw']
I0111 21:04:45.269598 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-67cb58cc7f-v69dw to start
I0111 21:04:48.502457 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-67cb58cc7f-v69dw ready, IP: 10.120.5.8
I0111 21:04:48.502723 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-67cb58cc7f-v69dw to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-67cb58cc7f-v69dw.log
I0111 21:04:48.503096 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-67cb58cc7f-v69dw
I0111 21:04:48.503246 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:04:48.601746 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:04:48.602331 140142540660736 client_app.py:244] [psm-grpc-client-67cb58cc7f-v69dw] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:04:48.700238 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:04:58.745362 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:08.789807 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:18.834194 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:28.878690 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:05:44.927716 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:09.975692 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:35.035231 140142540660736 client_app.py:371] [psm-grpc-client-67cb58cc7f-v69dw] Server channel:
I0111 21:06:35.067385 140142540660736 client_app.py:384] [psm-grpc-client-67cb58cc7f-v69dw] Found subchannel in state READY:
I0111 21:06:35.067677 140142540660736 client_app.py:255] [psm-grpc-client-67cb58cc7f-v69dw] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:06:35.067898 140142540660736 grpc.py:75] [psm-grpc-client-67cb58cc7f-v69dw:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:06:35.101624 140142540660736 grpc.py:75] [psm-grpc-client-67cb58cc7f-v69dw:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:06:39.111793 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-67cb58cc7f-v69dw] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-6cc98d756c-df447: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-6cc98d756c-df447: 100
I0111 21:06:39.112024 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') teardown -----
I0111 21:06:39.112339 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:06:49.374989 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:06:49.375309 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:06:50.144612 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:06:50.261777 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:06:50.262062 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:07:20.439661 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:07:20.507711 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:07:20.778751 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:07:20.779002 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:07:20.783113 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:07:21.691504 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007240785-60eb1eeed2edf-3dfb3df7-de009502
I0111 21:08:18.124012 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:08:28.299721 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:08:28.300086 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:08:28.398961 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:08:28.399232 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:08:28.948268 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:08:29.071103 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:08:29.071338 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:09:29.371460 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0')
I0111 21:09:29.372121 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version1 ('v0.13.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:203db6ce70452996f4183c30dd4c5ecaada168b0') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 21:09:29.372370 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.12.0 -----
I0111 21:09:29.374338 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:09:29.374613 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:09:29.374488Z%2F2024-01-11T21:39:29.374488Z?project=grpc-testing
I0111 21:09:29.376492 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:09:29.435277 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:09:29.438337 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:09:29.500017 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:09:29.500236 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:09:29.595968 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:09:29.596177 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:09:30.414870 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:09:30.471748 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:09:30.479581 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-8kg5wjth
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:09:30.552713 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:09:30.552945 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:09:30.657209 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-796b44865d-sgnlg']
I0111 21:09:30.657461 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-796b44865d-sgnlg to start
I0111 21:09:33.895037 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-796b44865d-sgnlg ready, IP: 10.120.2.185
I0111 21:09:33.895331 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-796b44865d-sgnlg to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-796b44865d-sgnlg.log
I0111 21:09:33.895732 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-796b44865d-sgnlg
I0111 21:09:33.895989 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:09:41.286038 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:09:41.323321 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:09:41.430749 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:09:41.431002 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:09:41.435462 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:09:42.386697 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007381437-60eb1f74f5cb7-e07f8328-a494f032
I0111 21:10:38.582781 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:10:38.852439 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:10:42.859508 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:10:42.859918 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:10:42.859783Z%2F2024-01-11T21:40:42.859783Z?project=grpc-testing
I0111 21:10:42.861672 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:10:42.905064 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:10:42.905291 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:10:43.529134 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:10:43.588904 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:10:43.597135 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ytbhtgub
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:10:43.667706 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:10:43.667982 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:10:43.766411 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7b7769fff5-tvbxn']
I0111 21:10:43.766671 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-7b7769fff5-tvbxn to start
I0111 21:10:47.002441 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-7b7769fff5-tvbxn ready, IP: 10.120.3.118
I0111 21:10:47.002708 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-7b7769fff5-tvbxn to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-7b7769fff5-tvbxn.log
I0111 21:10:47.003091 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-7b7769fff5-tvbxn
I0111 21:10:47.003233 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:10:47.097506 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:10:47.098055 140142540660736 client_app.py:244] [psm-grpc-client-7b7769fff5-tvbxn] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:10:47.197520 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:10:57.243497 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:07.288660 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:17.334598 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:27.379671 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:11:43.429177 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:08.489611 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:33.549543 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:58.609570 140142540660736 client_app.py:371] [psm-grpc-client-7b7769fff5-tvbxn] Server channel:
I0111 21:12:58.642272 140142540660736 client_app.py:384] [psm-grpc-client-7b7769fff5-tvbxn] Found subchannel in state READY:
I0111 21:12:58.642575 140142540660736 client_app.py:255] [psm-grpc-client-7b7769fff5-tvbxn] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:12:58.642813 140142540660736 grpc.py:75] [psm-grpc-client-7b7769fff5-tvbxn:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:12:58.677023 140142540660736 grpc.py:75] [psm-grpc-client-7b7769fff5-tvbxn:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:13:02.697348 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-7b7769fff5-tvbxn] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-796b44865d-sgnlg: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-796b44865d-sgnlg: 100
I0111 21:13:02.697618 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') teardown -----
I0111 21:13:02.697959 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:13:12.964744 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:13:12.965056 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:13:13.481553 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:13:13.595120 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:13:13.595342 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:13:43.776895 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:13:43.833095 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:13:44.208266 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:13:44.208482 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:13:44.212251 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:13:45.051813 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007624214-60eb205c7d609-01291ee3-fb94dfe6
I0111 21:14:39.532132 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:14:49.709480 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:14:49.709767 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:14:49.808210 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:14:49.808472 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:14:50.414696 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:14:50.530464 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:14:50.530708 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:16:50.949151 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2')
I0111 21:16:50.949785 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version2 ('v0.12.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:8765051ef3b742bc5cd20f16de078ae7547f2ba2') PASSED -----
[ RUN ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 21:16:50.950046 140142540660736 bootstrap_generator_test.py:202] ----- Testing bootstrap generator version v0.11.0 -----
I0111 21:16:50.952018 140142540660736 k8s_xds_server_runner.py:145] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20240111-2055-avpqw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0111 21:16:50.952279 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20240111-2055-avpqw%22;timeRange=2024-01-11T21:16:50.952160Z%2F2024-01-11T21:46:50.952160Z?project=grpc-testing
I0111 21:16:50.954123 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20240111-2055-avpqw
labels:
name: psm-interop-server-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:16:51.021661 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-server-20240111-2055-avpqw created
I0111 21:16:51.024875 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20240111-2055-avpqw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...
I0111 21:16:51.093547 140142540660736 k8s_base_runner.py:284] Service psm-grpc-server created
I0111 21:16:51.093785 140142540660736 k8s_base_runner.py:867] Waiting for 'cloud.google.com/neg-status' annotation for a NEG at port 8080 to be assigned to Kubernetes Service psm-grpc-server in namespace psm-interop-server-20240111-2055-avpqw
I0111 21:16:51.182155 140142540660736 k8s_base_runner.py:881] Detected 'cloud.google.com/neg-status' annotation for Kubernetes Service psm-grpc-server, namespace psm-interop-server-20240111-2055-avpqw: neg_name=psm-interop-server-20240111-2055-avpqw-psm-grpc-server, port=8080, zones=['us-central1-a']
I0111 21:16:51.182374 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:16:51.747051 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:16:51.804667 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-server created
I0111 21:16:51.812747 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20240111-2055-avpqw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-u7pb42t3
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/go-server:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata-experimental=app=psm-interop-server-20240111-2055-avpqw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:16:51.892325 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-server created
I0111 21:16:51.892588 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0111 21:16:52.002494 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-7657558f7-mr5vk']
I0111 21:16:52.002761 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-server-7657558f7-mr5vk to start
I0111 21:16:55.231371 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-server-7657558f7-mr5vk ready, IP: 10.120.5.10
I0111 21:16:55.231686 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-server-7657558f7-mr5vk to bootstrap_generator_test/test_app_logs/psm-interop-server-20240111-2055-avpqw_psm-grpc-server-7657558f7-mr5vk.log
I0111 21:16:55.232069 140142447158848 k8s_log_collector.py:70] Starting log collection thread 140142447158848 for psm-grpc-server-7657558f7-mr5vk
I0111 21:16:55.232215 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0111 21:17:10.928242 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-server has 1 replicas available
I0111 21:17:10.964773 140142540660736 traffic_director.py:256] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:17:11.089694 140142540660736 traffic_director.py:259] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:17:11.089963 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20240111-2055-avpqw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server', zone='us-central1-a')}
I0111 21:17:11.094450 140142540660736 compute.py:544] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20240111-2055-avpqw-psm-grpc-server
maxRatePerEndpoint: 5
...
I0111 21:17:11.895720 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705007831096-60eb2121c9cf9-d131e82e-b96d96f9
I0111 21:17:46.334476 140142540660736 traffic_director.py:294] Waiting for Backend Service psm-interop-backend-service-20240111-2055-avpqw to report all backends healthy: ['psm-interop-server-20240111-2055-avpqw-psm-grpc-server']
I0111 21:18:20.173809 140142540660736 compute.py:494] Backend psm-interop-server-20240111-2055-avpqw-psm-grpc-server in zone us-central1-a reported healthy
I0111 21:18:24.180645 140142540660736 k8s_xds_client_runner.py:111] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20240111-2055-avpqw: server_target=xds:///psm-grpc-server:4923 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0111 21:18:24.180991 140142540660736 k8s_base_runner.py:959] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20240111-2055-avpqw%22;timeRange=2024-01-11T21:18:24.180866Z%2F2024-01-11T21:48:24.180866Z?project=grpc-testing
I0111 21:18:24.182837 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20240111-2055-avpqw
labels:
name: psm-interop-client-20240111-2055-avpqw
owner: xds-k8s-interop-test
...
I0111 21:18:24.254984 140142540660736 k8s_base_runner.py:284] Namespace psm-interop-client-20240111-2055-avpqw created
I0111 21:18:24.255216 140142540660736 k8s_base_runner.py:373] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:18:24.831209 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...
I0111 21:18:24.893419 140142540660736 k8s_base_runner.py:284] ServiceAccount psm-grpc-client created
I0111 21:18:24.901731 140142540660736 k8s_base_runner.py:264] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20240111-2055-avpqw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-s8z61z6g
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/go-client:bb7b7332b803301b57f6497415512a9f5471fa0b
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:4923"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_ENABLE_OVERRIDE_HOST
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:7d8d90477792e2e1bfe3a3da20b3dc9ef01d326c
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...
I0111 21:18:24.970378 140142540660736 k8s_base_runner.py:284] Deployment psm-grpc-client created
I0111 21:18:24.970642 140142540660736 k8s_base_runner.py:797] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0111 21:18:25.071085 140142540660736 k8s_base_runner.py:807] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-57b79b84f-24k5b']
I0111 21:18:25.071403 140142540660736 k8s_base_runner.py:817] Waiting for pod psm-grpc-client-57b79b84f-24k5b to start
I0111 21:18:28.314275 140142540660736 k8s_base_runner.py:820] Pod psm-grpc-client-57b79b84f-24k5b ready, IP: 10.120.5.11
I0111 21:18:28.314561 140142540660736 k8s_base_runner.py:843] Enabling log collection from pod psm-grpc-client-57b79b84f-24k5b to bootstrap_generator_test/test_app_logs/psm-interop-client-20240111-2055-avpqw_psm-grpc-client-57b79b84f-24k5b.log
I0111 21:18:28.314958 140142366946880 k8s_log_collector.py:70] Starting log collection thread 140142366946880 for psm-grpc-client-57b79b84f-24k5b
I0111 21:18:28.315106 140142540660736 k8s_base_runner.py:779] Waiting for deployment psm-grpc-client to report 1 available replica(s)
I0111 21:18:29.519833 140142540660736 k8s_base_runner.py:788] Deployment psm-grpc-client has 1 replicas available
I0111 21:18:29.520420 140142540660736 client_app.py:244] [psm-grpc-client-57b79b84f-24k5b] Waiting to report a READY channel to xds:///psm-grpc-server:4923
I0111 21:18:29.619433 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:39.665026 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:49.710551 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:18:59.755653 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:09.796014 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:25.840476 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:50.875952 140142540660736 client_app.py:371] [psm-grpc-client-57b79b84f-24k5b] Server channel:
I0111 21:19:50.908937 140142540660736 client_app.py:384] [psm-grpc-client-57b79b84f-24k5b] Found subchannel in state READY:
I0111 21:19:50.909192 140142540660736 client_app.py:255] [psm-grpc-client-57b79b84f-24k5b] Channel to xds:///psm-grpc-server:4923 transitioned to state READY:
I0111 21:19:50.909413 140142540660736 grpc.py:75] [psm-grpc-client-57b79b84f-24k5b:8079] >> RPC ClientStatusDiscoveryService.FetchClientStatus(request=ClientStatusRequest({}), timeout=90, wait_for_ready=True)
I0111 21:19:50.943792 140142540660736 grpc.py:75] [psm-grpc-client-57b79b84f-24k5b:8079] >> RPC LoadBalancerStatsService.GetClientStats(request=LoadBalancerStatsRequest({'numRpcs': 100, 'timeoutSec': 1200}), timeout=1200, wait_for_ready=True)
I0111 21:19:54.966116 140142540660736 xds_k8s_testcase.py:601] [psm-grpc-client-57b79b84f-24k5b] << Received LoadBalancerStatsResponse:
num_failures: 0
rpcs_by_peer:
psm-grpc-server-7657558f7-mr5vk: 100
rpcs_by_method:
UnaryCall:
psm-grpc-server-7657558f7-mr5vk: 100
I0111 21:19:54.966348 140142540660736 bootstrap_generator_test.py:178] ----- TestMethod __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') teardown -----
I0111 21:19:54.966643 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-client
I0111 21:20:05.131322 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-client deleted
I0111 21:20:05.131700 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20240111-2055-avpqw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:20:05.715064 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-client
I0111 21:20:05.832372 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-client deleted
I0111 21:20:05.832652 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-client-20240111-2055-avpqw
I0111 21:20:36.068842 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-client-20240111-2055-avpqw deleted
I0111 21:20:36.131186 140142540660736 traffic_director.py:265] Waiting for Network Endpoint Groups to load endpoints.
I0111 21:20:36.406724 140142540660736 traffic_director.py:268] Loaded NEG "psm-interop-server-20240111-2055-avpqw-psm-grpc-server" in zone us-central1-a
I0111 21:20:36.406948 140142540660736 traffic_director.py:277] Adding backends to Backend Service psm-interop-backend-service-20240111-2055-avpqw: set()
I0111 21:20:36.410737 140142540660736 compute.py:544] Patching compute resource:
---
backends: []
...
I0111 21:20:37.424563 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008036412-60eb21e597dd3-a2232600-2f755c95
I0111 21:21:11.934650 140142540660736 k8s_base_runner.py:708] Deleting deployment psm-grpc-server
I0111 21:21:22.119263 140142540660736 k8s_base_runner.py:723] Deployment psm-grpc-server deleted
I0111 21:21:22.119597 140142540660736 k8s_base_runner.py:726] Deleting service psm-grpc-server
I0111 21:21:22.212309 140142540660736 k8s_base_runner.py:739] Service psm-grpc-server deleted
I0111 21:21:22.212624 140142540660736 k8s_base_runner.py:392] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20240111-2055-avpqw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0111 21:21:22.715710 140142540660736 k8s_base_runner.py:742] Deleting service account psm-grpc-server
I0111 21:21:22.831243 140142540660736 k8s_base_runner.py:756] Service account psm-grpc-server deleted
I0111 21:21:22.831511 140142540660736 k8s_base_runner.py:759] Deleting namespace psm-interop-server-20240111-2055-avpqw
I0111 21:23:23.257525 140142540660736 k8s_base_runner.py:776] Namespace psm-interop-server-20240111-2055-avpqw deleted
[ OK ] BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc')
I0111 21:23:23.258229 140142540660736 base_testcase.py:52] ----- TestCase __main__.BootstrapGeneratorServerTest.test_baseline_in_server_with_bootstrap_version3 ('v0.11.0', 'gcr.io/grpc-testing/td-grpc-bootstrap:b96f7a73314668aee83cbf86ab1e40135a0542fc') PASSED -----
I0111 21:23:23.258379 140142540660736 traffic_director.py:674] Deleting Forwarding rule "psm-interop-forwarding-rule-20240111-2055-avpqw"
I0111 21:23:23.903260 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008203262-60eb2284b6a87-f5089ad9-11c70026
I0111 21:23:36.784759 140142540660736 traffic_director.py:713] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20240111-2055-avpqw"
I0111 21:23:36.915439 140142540660736 traffic_director.py:605] Deleting HTTP Target proxy "psm-interop-target-proxy-20240111-2055-avpqw"
I0111 21:23:37.397751 140142540660736 traffic_director.py:593] Deleting Target GRPC proxy "psm-interop-target-proxy-20240111-2055-avpqw"
I0111 21:23:37.970142 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008217400-60eb229232699-37a0050d-cd5b443d
I0111 21:23:40.260673 140142540660736 traffic_director.py:633] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20240111-2055-avpqw"
I0111 21:23:40.649794 140142540660736 traffic_director.py:525] Deleting URL Map "psm-interop-url-map-20240111-2055-avpqw"
I0111 21:23:41.324169 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008220655-60eb22954cfd6-968197e3-87811ed9
I0111 21:23:45.848365 140142540660736 traffic_director.py:561] Deleting alternative URL Map "psm-interop-url-map-alt-20240111-2055-avpqw"
I0111 21:23:46.279969 140142540660736 traffic_director.py:243] Deleting Backend Service "psm-interop-backend-service-20240111-2055-avpqw"
I0111 21:23:46.929242 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008226285-60eb229aab6e2-e874402f-f6e61c3c
I0111 21:24:00.105541 140142540660736 traffic_director.py:332] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20240111-2055-avpqw"
I0111 21:24:00.452727 140142540660736 traffic_director.py:405] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20240111-2055-avpqw"
I0111 21:24:00.862147 140142540660736 traffic_director.py:202] Deleting Health Check "psm-interop-health-check-20240111-2055-avpqw"
I0111 21:24:01.376230 140142540660736 compute.py:614] Waiting 600 sec for compute operation id: operation-1705008240933-60eb22a8a3963-bdf30b90-40028f66
----------------------------------------------------------------------
Ran 8 tests in 2391.262s
OK
+ echo 'Failed test suites: 0'
Failed test suites: 0
[ID: 3109391] Command finished after 6588 secs, exit value: 0
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[13:24:06 PST] Collecting build artifacts from build VM
[13:24:07 PST] Kokoro builder finished