Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[00:00:54] Transferring environment variable script to build VM
[00:00:55] Transferring kokoro_log_reader.py to build VM
[00:00:55] Transferring source code to build VM
[00:00:56] Executing build script on build VM



[ID: 4659972] Executing command via SSH:
export KOKORO_BUILD_NUMBER="208"
export KOKORO_JOB_NAME="grpc/node/v1.8.x/xds_k8s_lb"
source /tmpfs/kokoro-env_vars.sh; cd /tmpfs/src/ ; chmod 755 github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; PYTHON_3_VERSION="$(pyenv which python3 2> /dev/null || which python3)" ; PYTHON_2_VERSION="$(pyenv which python2 2> /dev/null || which python2)" ; if "$PYTHON_3_VERSION" -c "import psutil" ; then KOKORO_PYTHON_COMMAND="$PYTHON_3_VERSION" ; else KOKORO_PYTHON_COMMAND="$PYTHON_2_VERSION" ; fi > /dev/null 2>&1 ; echo "export KOKORO_PYTHON_COMMAND="$KOKORO_PYTHON_COMMAND"" > "$HOME/.kokoro_python_vars" ; nohup bash -c "( rm -f /tmpfs/kokoro_build_exit_code ; github/grpc-node/packages/grpc-js-xds/scripts/xds_k8s_lb.sh ; echo \${PIPESTATUS[0]} > /tmpfs/kokoro_build_exit_code ) > /tmpfs/kokoro_build.log 2>&1" > /dev/null 2>&1 & echo $! > /tmpfs/kokoro_build.pid ; source "$HOME/.kokoro_python_vars" ; "$KOKORO_PYTHON_COMMAND" /tmpfs/kokoro_log_reader.py /tmpfs/kokoro_build.log /tmpfs/kokoro_build_exit_code /tmpfs/kokoro_build.pid /tmpfs/kokoro_log_reader.pid --start_byte 0

Submodule 'packages/grpc-js-xds/deps/envoy-api' (https://github.com/envoyproxy/data-plane-api.git) registered for path '../deps/envoy-api'
Submodule 'packages/grpc-js-xds/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../deps/googleapis'
Submodule 'packages/grpc-js-xds/deps/protoc-gen-validate' (https://github.com/envoyproxy/protoc-gen-validate.git) registered for path '../deps/protoc-gen-validate'
Submodule 'packages/grpc-js-xds/deps/xds' (https://github.com/cncf/xds.git) registered for path '../deps/xds'
Submodule 'packages/grpc-tools/deps/protobuf' (https://github.com/protocolbuffers/protobuf) registered for path '../../grpc-tools/deps/protobuf'
Submodule 'packages/proto-loader/deps/gapic-showcase' (https://github.com/googleapis/gapic-showcase.git) registered for path '../../proto-loader/deps/gapic-showcase'
Submodule 'packages/proto-loader/deps/googleapis' (https://github.com/googleapis/googleapis.git) registered for path '../../proto-loader/deps/googleapis'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/envoy-api'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/googleapis'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/protoc-gen-validate'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-js-xds/deps/xds'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase'...
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/googleapis'...
Submodule path '../deps/envoy-api': checked out '20b1b5fcee88a20a08b71051a961181839ec7268'
Submodule path '../deps/googleapis': checked out '8c53b2cb792234354c13336ac7daee61333deade'
Submodule path '../deps/protoc-gen-validate': checked out '0af61d9dc28712dc0e6f8e1a940855a2ee0cb9ed'
Submodule path '../deps/xds': checked out 'cb28da3451f158a947dfc45090fe92b07b243bc1'
Submodule path '../../grpc-tools/deps/protobuf': checked out '7c40b2df1fdf6f414c1c18c789715a9c948a0725'
Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path '../../grpc-tools/deps/protobuf/third_party/benchmark'
Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path '../../grpc-tools/deps/protobuf/third_party/googletest'
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/benchmark'...
Cloning into '/tmpfs/src/github/grpc-node/packages/grpc-tools/deps/protobuf/third_party/googletest'...
Submodule path '../../grpc-tools/deps/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
Submodule path '../../grpc-tools/deps/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
Submodule path '../../proto-loader/deps/gapic-showcase': checked out 'b09b3ba9a8db8aae7d5d7c3939853681cc97c293'
Submodule 'schema/api-common-protos' (https://github.com/googleapis/api-common-protos.git) registered for path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos'
Cloning into '/tmpfs/src/github/grpc-node/packages/proto-loader/deps/gapic-showcase/schema/api-common-protos'...
Submodule path '../../proto-loader/deps/gapic-showcase/schema/api-common-protos': checked out '4c0a203e3658ae0e56d47c817c2c5904116c0ae0'
Submodule path '../../proto-loader/deps/googleapis': checked out '8f2eda119e11c8bd0c189b545da18bba9019c83e'
Sourcing test driver install script from: https://raw.githubusercontent.com/grpc/grpc/master/tools/internal_ci/linux/grpc_xds_k8s_install_test_driver.sh
Activated GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
Activated secondary GKE cluster: GKE_CLUSTER_NAME=psm-interop-lb-primary GKE_CLUSTER_ZONE=us-central1-a
+ [[ -n /tmpfs/src ]]
+ kokoro_setup_test_driver grpc-node
+ local src_repository_name=grpc-node
+ kokoro_print_version
+ echo 'Kokoro VM version:'
Kokoro VM version:
+ [[ -f /VERSION ]]
+ cat /VERSION
kokoro-ubuntu1604-v2b
+ run_ignore_exit_code lsb_release -a
+ local exit_code=-1
+ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.04.7 LTS
Release: 16.04
Codename: xenial
+ echo 'Exit code: -1'
Exit code: -1
+ kokoro_get_testing_version
+ local version_from_job_name
++ echo grpc/node/v1.8.x/xds_k8s_lb
++ cut -d / -f3
+ version_from_job_name=v1.8.x
+ [[ -n '' ]]
+ [[ sergiitk != \k\o\k\o\r\o ]]
+ [[ -n v1.8.x ]]
+ readonly TESTING_VERSION=dev-v1.8.x
+ TESTING_VERSION=dev-v1.8.x
+ local github_root=/tmpfs/src/github
+ readonly SRC_DIR=/tmpfs/src/github/grpc-node
+ SRC_DIR=/tmpfs/src/github/grpc-node
+ local test_driver_repo_dir
++ mktemp -d
+ test_driver_repo_dir=/tmpfs/tmp/tmp.OvLKtVktid/grpc
+ parse_src_repo_git_info SRC_DIR
+ local src_dir=/tmpfs/src/github/grpc-node
++ git -C /tmpfs/src/github/grpc-node remote get-url origin
+ readonly GIT_ORIGIN_URL=https://github.com/murgatroid99/grpc-node.git
+ GIT_ORIGIN_URL=https://github.com/murgatroid99/grpc-node.git
++ git -C /tmpfs/src/github/grpc-node rev-parse HEAD
+ readonly GIT_COMMIT=9441de78f655ada34ada0dc1a8057122eb21f229
+ GIT_COMMIT=9441de78f655ada34ada0dc1a8057122eb21f229
++ git -C /tmpfs/src/github/grpc-node rev-parse --short HEAD
+ readonly GIT_COMMIT_SHORT=9441de78
+ GIT_COMMIT_SHORT=9441de78
+ kokoro_write_sponge_properties
+ cat
+ echo 'Sponge properties:'
Sponge properties:
+ cat /tmpfs/src/custom_sponge_config.csv
TESTS_FORMAT_VERSION,2
TESTGRID_EXCLUDE,1
GIT_ORIGIN_URL,https://github.com/murgatroid99/grpc-node.git
GIT_COMMIT_SHORT,9441de78
+ kokoro_setup_python_virtual_environment
+ echo 'Setup pyenv environment'
Setup pyenv environment
++ pyenv init -
+ eval 'export PYENV_SHELL=bash
source '\''/home/kbuilder/.pyenv/libexec/../completions/pyenv.bash'\''
command pyenv rehash 2>/dev/null
pyenv() {
local command
command="${1:-}"
if [ "$#" -gt 0 ]; then
shift
fi

case "$command" in
activate|deactivate|rehash|shell)
eval "$(pyenv "sh-$command" "$@")"
;;
*)
command pyenv "$command" "$@"
;;
esac
}'
++ export PYENV_SHELL=bash
++ PYENV_SHELL=bash
++ source /home/kbuilder/.pyenv/libexec/../completions/pyenv.bash
+++ complete -F _pyenv pyenv
++ command pyenv rehash
++ pyenv virtualenv-init -
++ local command
++ command=virtualenv-init
++ '[' 2 -gt 0 ']'
++ shift
++ case "$command" in
++ command pyenv virtualenv-init -
++ pyenv virtualenv-init -
+ eval 'export PATH="/home/kbuilder/.pyenv/plugins/pyenv-virtualenv/shims:${PATH}";
export PYENV_VIRTUALENV_INIT=1;
_pyenv_virtualenv_hook() {
local ret=$?
if [ -n "$VIRTUAL_ENV" ]; then
eval "$(pyenv sh-activate --quiet || pyenv sh-deactivate --quiet || true)" || true
else
eval "$(pyenv sh-activate --quiet || true)" || true
fi
return $ret
};
if ! [[ "$PROMPT_COMMAND" =~ _pyenv_virtualenv_hook ]]; then
PROMPT_COMMAND="_pyenv_virtualenv_hook;$PROMPT_COMMAND";
fi'
++ export PATH=/home/kbuilder/.pyenv/plugins/pyenv-virtualenv/shims:/opt/google-cloud-sdk/bin:/home/kbuilder/.pyenv/plugins/pyenv-virtualenv/shims:/home/kbuilder/.pyenv/shims:/home/kbuilder/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/usr/local/rvm/bin:/opt/android-sdk/current/bin:/usr/local/go/bin:/usr/local/go/packages/bin:/opt/android-sdk/current/bin:/usr/local/go/bin:/usr/local/go/packages/bin:/opt/kubernetes/client/bin:/opt/android-studio/bin:/usr/local/apache-maven/bin
++ PATH=/home/kbuilder/.pyenv/plugins/pyenv-virtualenv/shims:/opt/google-cloud-sdk/bin:/home/kbuilder/.pyenv/plugins/pyenv-virtualenv/shims:/home/kbuilder/.pyenv/shims:/home/kbuilder/.pyenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/usr/local/rvm/bin:/opt/android-sdk/current/bin:/usr/local/go/bin:/usr/local/go/packages/bin:/opt/android-sdk/current/bin:/usr/local/go/bin:/usr/local/go/packages/bin:/opt/kubernetes/client/bin:/opt/android-studio/bin:/usr/local/apache-maven/bin
++ export PYENV_VIRTUALENV_INIT=1
++ PYENV_VIRTUALENV_INIT=1
++ [[ '' =~ _pyenv_virtualenv_hook ]]
++ PROMPT_COMMAND='_pyenv_virtualenv_hook;'
++ pyenv versions --bare --skip-aliases
++ grep -E '^3.9\.[0-9]{1,2}$'
++ local command
++ command=versions
++ '[' 3 -gt 0 ']'
++ sort --version-sort
++ shift
++ case "$command" in
++ command pyenv versions --bare --skip-aliases
++ pyenv versions --bare --skip-aliases
++ tail -n 1
+ py_latest_patch=3.9.5
+ echo 'Activating python 3.9.5 virtual environment'
Activating python 3.9.5 virtual environment
+ pyenv virtualenv --without-pip 3.9.5 k8s_xds_test_runner
+ local command
+ command=virtualenv
+ '[' 4 -gt 0 ']'
+ shift
+ case "$command" in
+ command pyenv virtualenv --without-pip 3.9.5 k8s_xds_test_runner
+ pyenv virtualenv --without-pip 3.9.5 k8s_xds_test_runner
+ pyenv local k8s_xds_test_runner
+ local command
+ command=local
+ '[' 2 -gt 0 ']'
+ shift
+ case "$command" in
+ command pyenv local k8s_xds_test_runner
+ pyenv local k8s_xds_test_runner
+ pyenv activate k8s_xds_test_runner
+ local command
+ command=activate
+ '[' 2 -gt 0 ']'
+ shift
+ case "$command" in
++ pyenv sh-activate k8s_xds_test_runner
++ local command
++ command=sh-activate
++ '[' 2 -gt 0 ']'
++ shift
++ case "$command" in
++ command pyenv sh-activate k8s_xds_test_runner
++ pyenv sh-activate k8s_xds_test_runner
pyenv-virtualenv: prompt changing will be removed from future release. configure `export PYENV_VIRTUALENV_DISABLE_PROMPT=1' to simulate the behavior.
+ eval 'unset PYENV_VIRTUAL_ENV;
unset VIRTUAL_ENV;
if [ -n "${_OLD_VIRTUAL_PATH}" ]; then
export PATH="${_OLD_VIRTUAL_PATH}";
unset _OLD_VIRTUAL_PATH;
fi;
if [ -n "${_OLD_VIRTUAL_PYTHONHOME}" ]; then
export PYTHONHOME="${_OLD_VIRTUAL_PYTHONHOME}";
unset _OLD_VIRTUAL_PYTHONHOME;
fi;
if [ -n "${_OLD_VIRTUAL_PS1}" ]; then
export PS1="${_OLD_VIRTUAL_PS1}";
unset _OLD_VIRTUAL_PS1;
fi;
if declare -f deactivate 1>/dev/null 2>&1; then
unset -f deactivate;
fi;
export PYENV_VERSION="k8s_xds_test_runner";
export PYENV_ACTIVATE_SHELL=1;
export PYENV_VIRTUAL_ENV="/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner";
export VIRTUAL_ENV="/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner";
export _OLD_VIRTUAL_PS1="${PS1}";
export PS1="(k8s_xds_test_runner) ${PS1}";'
++ unset PYENV_VIRTUAL_ENV
++ unset VIRTUAL_ENV
++ '[' -n '' ']'
++ '[' -n '' ']'
++ '[' -n '' ']'
++ declare -f deactivate
++ export PYENV_VERSION=k8s_xds_test_runner
++ PYENV_VERSION=k8s_xds_test_runner
++ export PYENV_ACTIVATE_SHELL=1
++ PYENV_ACTIVATE_SHELL=1
++ export PYENV_VIRTUAL_ENV=/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner
++ PYENV_VIRTUAL_ENV=/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner
++ export VIRTUAL_ENV=/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner
++ VIRTUAL_ENV=/home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner
++ export _OLD_VIRTUAL_PS1=
++ _OLD_VIRTUAL_PS1=
++ export 'PS1=(k8s_xds_test_runner) '
++ PS1='(k8s_xds_test_runner) '
+ python3 -m ensurepip
Looking in links: /tmpfs/tmp/tmpx7rmplhi
Processing /tmpfs/tmp/tmpx7rmplhi/setuptools-56.0.0-py3-none-any.whl
Processing /tmpfs/tmp/tmpx7rmplhi/pip-21.1.1-py3-none-any.whl
Installing collected packages: setuptools, pip
Successfully installed pip-21.1.1 setuptools-56.0.0
+ python3 -m pip install -U pip==21.0.1
Collecting pip==21.0.1
Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
Attempting uninstall: pip
Found existing installation: pip 21.1.1
Uninstalling pip-21.1.1:
Successfully uninstalled pip-21.1.1
Successfully installed pip-21.0.1
+ python3 -m pip --version
pip 21.0.1 from /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/pip (python 3.9)
+ gcloud_update
+ echo 'Update gcloud components:'
Update gcloud components:
+ gcloud -q components update


Your current Cloud SDK version is: 235.0.0
You will be upgraded to version: 436.0.0

+----------------------------------------------------------------------------+
| These components will be removed. |
+-----------------------------------------------------+---------------+------+
| Name | Version | Size |
+-----------------------------------------------------+---------------+------+
| Cloud Datalab Command Line Tool | 20190116 | |
| Cloud Datalab Command Line Tool (Platform Specific) | 20180823 | |
| Cloud Datastore Emulator (Legacy) | v1beta3-1.0.0 | |
| Emulator Reverse Proxy | | |
| Emulator Reverse Proxy | 0.0.1 | |
+-----------------------------------------------------+---------------+------+
+------------------------------------------------------------------------------+
| These components will be updated. |
+-----------------------------------------------------+------------+-----------+
| Name | Version | Size |
+-----------------------------------------------------+------------+-----------+
| App Engine Go Extensions | 1.9.75 | 4.5 MiB |
| BigQuery Command Line Tool | 2.0.93 | 1.6 MiB |
| BigQuery Command Line Tool (Platform Specific) | 2.0.77 | < 1 MiB |
| Cloud Bigtable Command Line Tool | 0.16.0 | 11.4 MiB |
| Cloud Bigtable Emulator | | 7.0 MiB |
| Cloud Datastore Emulator | 2.3.0 | 35.1 MiB |
| Cloud Firestore Emulator | 1.18.1 | 42.5 MiB |
| Cloud Pub/Sub Emulator | 0.8.2 | 62.6 MiB |
| Cloud SQL Proxy | 1.27.0 | 7.8 MiB |
| Cloud Storage Command Line Tool | 5.24 | 11.3 MiB |
| Cloud Storage Command Line Tool (Platform Specific) | 5.13 | < 1 MiB |
| Google Cloud Build Local Builder | 0.5.2 | 6.3 MiB |
| Google Cloud CLI Core Libraries | 2023.06.16 | 20.9 MiB |
| Google Cloud CLI Core Libraries (Platform Specific) | 2022.09.20 | < 1 MiB |
| gRPC Python library | 1.20.0 | 2.1 MiB |
| gcloud Alpha Commands | 2023.06.16 | < 1 MiB |
| gcloud Beta Commands | 2023.06.16 | < 1 MiB |
| gcloud app Java Extensions | 2.0.14 | 64.6 MiB |
| gcloud app Python Extensions | 1.9.104 | 8.5 MiB |
| gcloud app Python Extensions (Extra Libraries) | 1.9.100 | 27.3 MiB |
| gcloud cli dependencies | 2023.06.09 | 11.0 MiB |
| gcloud cli dependencies | 2021.04.16 | < 1 MiB |
| kubectl | 1.25.9 | 107.2 MiB |
| kubectl | 1.25.9 | < 1 MiB |
+-----------------------------------------------------+------------+-----------+
+---------------------------------------------+
| These components will be installed. |
+------------------------+---------+----------+
| Name | Version | Size |
+------------------------+---------+----------+
| Bundled Python 3.9 | 3.9.16 | 63.5 MiB |
| Bundled Python 3.9 | 3.9.16 | |
| gke-gcloud-auth-plugin | 0.5.3 | |
| gke-gcloud-auth-plugin | 0.5.3 | 7.7 MiB |
+------------------------+---------+----------+

A lot has changed since your last upgrade. For the latest full release notes,
please visit:
https://cloud.google.com/sdk/release_notes

#============================================================#
#= Creating update staging area =#
#============================================================#
#= Uninstalling: App Engine Go Extensions =#
#============================================================#
#= Uninstalling: BigQuery Command Line Tool =#
#============================================================#
#= Uninstalling: BigQuery Command Line Tool (Platform Sp... =#
#============================================================#
#= Uninstalling: Cloud Bigtable Command Line Tool =#
#============================================================#
#= Uninstalling: Cloud Bigtable Emulator =#
#============================================================#
#= Uninstalling: Cloud Datalab Command Line Tool =#
#============================================================#
#= Uninstalling: Cloud Datalab Command Line Tool (Platfo... =#
#============================================================#
#= Uninstalling: Cloud Datastore Emulator =#
#============================================================#
#= Uninstalling: Cloud Datastore Emulator (Legacy) =#
#============================================================#
#= Uninstalling: Cloud Firestore Emulator =#
#============================================================#
#= Uninstalling: Cloud Pub/Sub Emulator =#
#============================================================#
#= Uninstalling: Cloud SDK Core Libraries =#
#============================================================#
#= Uninstalling: Cloud SDK Core Libraries (Platform Spec... =#
#============================================================#
#= Uninstalling: Cloud SQL Proxy =#
#============================================================#
#= Uninstalling: Cloud Storage Command Line Tool =#
#============================================================#
#= Uninstalling: Cloud Storage Command Line Tool (Platfo... =#
#============================================================#
#= Uninstalling: Emulator Reverse Proxy =#
#============================================================#
#= Uninstalling: Emulator Reverse Proxy =#
#============================================================#
#= Uninstalling: Google Cloud Build Local Builder =#
#============================================================#
#= Uninstalling: gRPC python library =#
#============================================================#
#= Uninstalling: gcloud Alpha Commands =#
#============================================================#
#= Uninstalling: gcloud Beta Commands =#
#============================================================#
#= Uninstalling: gcloud app Java Extensions =#
#============================================================#
#= Uninstalling: gcloud app Python Extensions =#
#============================================================#
#= Uninstalling: gcloud app Python Extensions (Extra Lib... =#
#============================================================#
#= Uninstalling: gcloud cli dependencies =#
#============================================================#
#= Uninstalling: gcloud cli dependencies =#
#============================================================#
#= Uninstalling: kubectl =#
#============================================================#
#= Uninstalling: kubectl =#
#============================================================#
#= Installing: App Engine Go Extensions =#
#============================================================#
#= Installing: BigQuery Command Line Tool =#
#============================================================#
#= Installing: BigQuery Command Line Tool (Platform Spec... =#
#============================================================#
#= Installing: Bundled Python 3.9 =#
#============================================================#
#= Installing: Bundled Python 3.9 =#
#============================================================#
#= Installing: Cloud Bigtable Command Line Tool =#
#============================================================#
#= Installing: Cloud Bigtable Emulator =#
#============================================================#
#= Installing: Cloud Datastore Emulator =#
#============================================================#
#= Installing: Cloud Firestore Emulator =#
#============================================================#
#= Installing: Cloud Pub/Sub Emulator =#
#============================================================#
#= Installing: Cloud SQL Proxy =#
#============================================================#
#= Installing: Cloud Storage Command Line Tool =#
#============================================================#
#= Installing: Cloud Storage Command Line Tool (Platform... =#
#============================================================#
#= Installing: Google Cloud Build Local Builder =#
#============================================================#
#= Installing: Google Cloud CLI Core Libraries =#
#============================================================#
#= Installing: Google Cloud CLI Core Libraries (Platform... =#
#============================================================#
#= Installing: gRPC Python library =#
#============================================================#
#= Installing: gcloud Alpha Commands =#
#============================================================#
#= Installing: gcloud Beta Commands =#
#============================================================#
#= Installing: gcloud app Java Extensions =#
#============================================================#
#= Installing: gcloud app Python Extensions =#
#============================================================#
#= Installing: gcloud app Python Extensions (Extra Libra... =#
#============================================================#
#= Installing: gcloud cli dependencies =#
#============================================================#
#= Installing: gcloud cli dependencies =#
#============================================================#
#= Installing: gke-gcloud-auth-plugin =#
#============================================================#
#= Installing: gke-gcloud-auth-plugin =#
#============================================================#
#= Installing: kubectl =#
#============================================================#
#= Installing: kubectl =#
#============================================================#
#= Creating backup and activating new installation =#
#============================================================#

Performing post processing steps...
failed.
WARNING: Post processing failed. Run `gcloud info --show-log` to view the failures.

Update done!

To revert your SDK to the previously installed version, you may run:
$ gcloud components update --version 235.0.0

WARNING: There are older versions of Google Cloud Platform tools on your system PATH.
Please remove the following to avoid accidentally invoking these old tools:

/opt/kubernetes/client/bin/kubectl
/usr/bin/kubectl


+ gcloud_get_cluster_credentials
+ [[ -n psm-interop-lb-secondary ]]
+ [[ -n us-west1-b ]]
+ gcloud container clusters get-credentials psm-interop-lb-secondary --zone us-west1-b
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-secondary.
++ kubectl config current-context
+ readonly SECONDARY_KUBE_CONTEXT=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
+ SECONDARY_KUBE_CONTEXT=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary
+ gcloud container clusters get-credentials psm-interop-lb-primary --zone us-central1-a
Fetching cluster endpoint and auth data.
kubeconfig entry generated for psm-interop-lb-primary.
++ kubectl config current-context
+ readonly KUBE_CONTEXT=gke_grpc-testing_us-central1-a_psm-interop-lb-primary
+ KUBE_CONTEXT=gke_grpc-testing_us-central1-a_psm-interop-lb-primary
+ test_driver_install /tmpfs/tmp/tmp.OvLKtVktid/grpc
+ readonly TEST_DRIVER_REPO_DIR=/tmpfs/tmp/tmp.OvLKtVktid/grpc
+ TEST_DRIVER_REPO_DIR=/tmpfs/tmp/tmp.OvLKtVktid/grpc
+ readonly TEST_DRIVER_FULL_DIR=/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver
+ TEST_DRIVER_FULL_DIR=/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver
+ test_driver_get_source
+ [[ -n '' ]]
+ echo 'Cloning driver to https://github.com/grpc/grpc.git branch master to /tmpfs/tmp/tmp.OvLKtVktid/grpc'
Cloning driver to https://github.com/grpc/grpc.git branch master to /tmpfs/tmp/tmp.OvLKtVktid/grpc
+ git clone -b master --depth=1 https://github.com/grpc/grpc.git /tmpfs/tmp/tmp.OvLKtVktid/grpc
Cloning into '/tmpfs/tmp/tmp.OvLKtVktid/grpc'...
+ test_driver_pip_install
+ echo 'Install python dependencies'
Install python dependencies
+ cd /tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver
+ [[ -z /home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner ]]
+ python3 -m pip install -r requirements.lock
Collecting Mako==1.2.4
Downloading Mako-1.2.4-py3-none-any.whl (78 kB)
Collecting PyYAML==5.4.1
Downloading PyYAML-5.4.1-cp39-cp39-manylinux1_x86_64.whl (630 kB)
Collecting absl-py==0.15.0
Downloading absl_py-0.15.0-py3-none-any.whl (132 kB)
Collecting google-api-python-client==1.12.11
Downloading google_api_python_client-1.12.11-py2.py3-none-any.whl (62 kB)
Collecting google-cloud-secret-manager==2.15.1
Downloading google_cloud_secret_manager-2.15.1-py2.py3-none-any.whl (100 kB)
Collecting grpcio==1.51.1
Downloading grpcio-1.51.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
Collecting grpcio-health-checking==1.48.2
Downloading grpcio_health_checking-1.48.2-py3-none-any.whl (18 kB)
Collecting grpcio-tools==1.48.2
Downloading grpcio_tools-1.48.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.4 MB)
Collecting grpcio-channelz==1.48.2
Downloading grpcio_channelz-1.48.2-py3-none-any.whl (21 kB)
Collecting kubernetes==25.3.0
Downloading kubernetes-25.3.0-py2.py3-none-any.whl (1.4 MB)
Collecting six==1.16.0
Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting tenacity==6.3.1
Downloading tenacity-6.3.1-py2.py3-none-any.whl (36 kB)
Collecting packaging==21.3
Downloading packaging-21.3-py3-none-any.whl (40 kB)
Collecting Pygments==2.14.0
Downloading Pygments-2.14.0-py3-none-any.whl (1.1 MB)
Collecting python-dateutil==2.8.2
Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
Collecting protobuf==3.20.3
Downloading protobuf-3.20.3-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.0 MB)
Collecting xds-protos==0.0.11
Downloading xds_protos-0.0.11-py3-none-any.whl (1.8 MB)
Collecting cachetools==5.3.0
Downloading cachetools-5.3.0-py3-none-any.whl (9.3 kB)
Collecting certifi==2022.12.7
Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)
Collecting charset-normalizer==3.0.1
Downloading charset_normalizer-3.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (198 kB)
Collecting google-api-core==2.11.0
Downloading google_api_core-2.11.0-py3-none-any.whl (120 kB)
Collecting google-auth==2.16.0
Downloading google_auth-2.16.0-py2.py3-none-any.whl (177 kB)
Collecting google-auth-httplib2==0.1.0
Downloading google_auth_httplib2-0.1.0-py2.py3-none-any.whl (9.3 kB)
Collecting googleapis-common-protos==1.58.0
Downloading googleapis_common_protos-1.58.0-py2.py3-none-any.whl (223 kB)
Collecting grpc-google-iam-v1==0.12.6
Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)
Collecting grpcio-status==1.48.2
Downloading grpcio_status-1.48.2-py3-none-any.whl (14 kB)
Collecting httplib2==0.21.0
Downloading httplib2-0.21.0-py3-none-any.whl (96 kB)
Collecting idna==3.4
Downloading idna-3.4-py3-none-any.whl (61 kB)
Collecting MarkupSafe==2.1.2
Downloading MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting oauthlib==3.2.2
Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)
Collecting proto-plus==1.22.2
Downloading proto_plus-1.22.2-py3-none-any.whl (47 kB)
Collecting pyasn1==0.4.8
Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting pyasn1-modules==0.2.8
Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyparsing==3.0.9
Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)
Collecting requests==2.28.2
Downloading requests-2.28.2-py3-none-any.whl (62 kB)
Collecting requests-oauthlib==1.3.1
Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
Collecting rsa==4.9
Downloading rsa-4.9-py3-none-any.whl (34 kB)
Collecting uritemplate==3.0.1
Downloading uritemplate-3.0.1-py2.py3-none-any.whl (15 kB)
Collecting urllib3==1.26.14
Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)
Collecting websocket-client==1.5.1
Downloading websocket_client-1.5.1-py3-none-any.whl (55 kB)
Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.10.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,!=2.8.*,!=2.9.*,<3.0.0dev,>=1.34.0
Downloading google_api_core-2.11.1-py3-none-any.whl (120 kB)
Collecting googleapis-common-protos[grpc]<2.0.0dev,>=1.56.0
Downloading googleapis_common_protos-1.59.1-py2.py3-none-any.whl (224 kB)
Requirement already satisfied: setuptools in /home/kbuilder/.pyenv/versions/3.9.5/envs/k8s_xds_test_runner/lib/python3.9/site-packages (from grpcio-tools==1.48.2->-r requirements.lock (line 8)) (56.0.0)
Downloading googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
Installing collected packages: pyasn1, urllib3, six, rsa, pyasn1-modules, protobuf, idna, charset-normalizer, certifi, cachetools, requests, pyparsing, grpcio, googleapis-common-protos, google-auth, oauthlib, httplib2, grpcio-status, google-api-core, websocket-client, uritemplate, requests-oauthlib, PyYAML, python-dateutil, proto-plus, MarkupSafe, grpc-google-iam-v1, google-auth-httplib2, xds-protos, tenacity, Pygments, packaging, Mako, kubernetes, grpcio-tools, grpcio-health-checking, grpcio-channelz, google-cloud-secret-manager, google-api-python-client, absl-py
Successfully installed Mako-1.2.4 MarkupSafe-2.1.2 PyYAML-5.4.1 Pygments-2.14.0 absl-py-0.15.0 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.0.1 google-api-core-2.11.0 google-api-python-client-1.12.11 google-auth-2.16.0 google-auth-httplib2-0.1.0 google-cloud-secret-manager-2.15.1 googleapis-common-protos-1.58.0 grpc-google-iam-v1-0.12.6 grpcio-1.51.1 grpcio-channelz-1.48.2 grpcio-health-checking-1.48.2 grpcio-status-1.48.2 grpcio-tools-1.48.2 httplib2-0.21.0 idna-3.4 kubernetes-25.3.0 oauthlib-3.2.2 packaging-21.3 proto-plus-1.22.2 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 python-dateutil-2.8.2 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 six-1.16.0 tenacity-6.3.1 uritemplate-3.0.1 urllib3-1.26.14 websocket-client-1.5.1 xds-protos-0.0.11
WARNING: You are using pip version 21.0.1; however, version 23.1.2 is available.
You should consider upgrading via the '/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3 -m pip install --upgrade pip' command.
+ echo 'Installed Python packages:'
Installed Python packages:
+ python3 -m pip list
Package Version
--------------------------- ---------
absl-py 0.15.0
cachetools 5.3.0
certifi 2022.12.7
charset-normalizer 3.0.1
google-api-core 2.11.0
google-api-python-client 1.12.11
google-auth 2.16.0
google-auth-httplib2 0.1.0
google-cloud-secret-manager 2.15.1
googleapis-common-protos 1.58.0
grpc-google-iam-v1 0.12.6
grpcio 1.51.1
grpcio-channelz 1.48.2
grpcio-health-checking 1.48.2
grpcio-status 1.48.2
grpcio-tools 1.48.2
httplib2 0.21.0
idna 3.4
kubernetes 25.3.0
Mako 1.2.4
MarkupSafe 2.1.2
oauthlib 3.2.2
packaging 21.3
pip 21.0.1
proto-plus 1.22.2
protobuf 3.20.3
pyasn1 0.4.8
pyasn1-modules 0.2.8
Pygments 2.14.0
pyparsing 3.0.9
python-dateutil 2.8.2
PyYAML 5.4.1
requests 2.28.2
requests-oauthlib 1.3.1
rsa 4.9
setuptools 56.0.0
six 1.16.0
tenacity 6.3.1
uritemplate 3.0.1
urllib3 1.26.14
websocket-client 1.5.1
xds-protos 0.0.11
WARNING: You are using pip version 21.0.1; however, version 23.1.2 is available.
You should consider upgrading via the '/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3 -m pip install --upgrade pip' command.
+ test_driver_compile_protos
+ declare -a protos
+ protos=("${TEST_DRIVER_PROTOS_PATH}/test.proto" "${TEST_DRIVER_PROTOS_PATH}/messages.proto" "${TEST_DRIVER_PROTOS_PATH}/empty.proto")
+ echo 'Generate python code from grpc.testing protos: src/proto/grpc/testing/test.proto src/proto/grpc/testing/messages.proto src/proto/grpc/testing/empty.proto'
Generate python code from grpc.testing protos: src/proto/grpc/testing/test.proto src/proto/grpc/testing/messages.proto src/proto/grpc/testing/empty.proto
+ cd /tmpfs/tmp/tmp.OvLKtVktid/grpc
+ python3 -m grpc_tools.protoc --proto_path=. --python_out=/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver --grpc_python_out=/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver src/proto/grpc/testing/test.proto src/proto/grpc/testing/messages.proto src/proto/grpc/testing/empty.proto
+ local protos_out_dir=/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing
+ echo 'Generated files /tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing:'
Generated files /tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing:
+ ls -Fl /tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/src/proto/grpc/testing
total 84
-rw-rw-r-- 1 kbuilder kbuilder 159 Jun 23 00:03 empty_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 1163 Jun 23 00:03 empty_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 159 Jun 23 00:03 messages_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 29462 Jun 23 00:03 messages_pb2.py
-rw-rw-r-- 1 kbuilder kbuilder 34950 Jun 23 00:03 test_pb2_grpc.py
-rw-rw-r-- 1 kbuilder kbuilder 3934 Jun 23 00:03 test_pb2.py
+ readonly TEST_DRIVER_FLAGFILE=config/grpc-testing.cfg
+ TEST_DRIVER_FLAGFILE=config/grpc-testing.cfg
+ local artifacts_dir=/tmpfs/src/artifacts
+ readonly TEST_XML_OUTPUT_DIR=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb
+ TEST_XML_OUTPUT_DIR=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb
+ mkdir -p /tmpfs/src/artifacts /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb
+ build_docker_images_if_needed
++ gcloud_gcr_list_image_tags gcr.io/grpc-testing/xds-interop/node-client 9441de78f655ada34ada0dc1a8057122eb21f229
++ gcloud container images list-tags '--format=table[box](tags,digest,timestamp.date())' --filter=tags:9441de78f655ada34ada0dc1a8057122eb21f229 gcr.io/grpc-testing/xds-interop/node-client
+ client_tags=
+ printf 'Client image: %s:%s\n' gcr.io/grpc-testing/xds-interop/node-client 9441de78f655ada34ada0dc1a8057122eb21f229
Client image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
+ echo 'Client image not found'
Client image not found
+ [[ 0 == \1 ]]
+ [[ -z '' ]]
+ build_test_app_docker_images
+ echo 'Building Node xDS interop test app Docker images'
Building Node xDS interop test app Docker images
+ pushd /tmpfs/src/github/grpc-node
/tmpfs/src/github/grpc-node /tmpfs/tmp/tmp.OvLKtVktid/grpc
+ docker build -f packages/grpc-js-xds/interop/Dockerfile -t gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 .
Sending build context to Docker daemon 685.4MB
Step 1/14 : FROM node:18-slim as build
18-slim: Pulling from library/node
5b5fe70539cd: Pulling fs layer
b6b4ba8b66ef: Pulling fs layer
f39e6a5b2145: Pulling fs layer
4ab682218b63: Pulling fs layer
8e4ca3aa328d: Pulling fs layer
4ab682218b63: Waiting
8e4ca3aa328d: Waiting
b6b4ba8b66ef: Download complete
5b5fe70539cd: Verifying Checksum
5b5fe70539cd: Download complete
4ab682218b63: Verifying Checksum
f39e6a5b2145: Verifying Checksum
f39e6a5b2145: Download complete
8e4ca3aa328d: Verifying Checksum
8e4ca3aa328d: Download complete
5b5fe70539cd: Pull complete
b6b4ba8b66ef: Pull complete
f39e6a5b2145: Pull complete
4ab682218b63: Pull complete
8e4ca3aa328d: Pull complete
Digest: sha256:3a57143aa02727b39b72a6242e1b25df1998c601aa754a6dff9ce8936dc4f5a6
Status: Downloaded newer image for node:18-slim
---> 0fbd8cae03d7
Step 2/14 : WORKDIR /node/src/grpc-node
---> Running in 274e5e032ec8
Removing intermediate container 274e5e032ec8
---> a4fb56040a96
Step 3/14 : COPY . .
---> 5d0e0b35235f
Step 4/14 : WORKDIR /node/src/grpc-node/packages/grpc-js
---> Running in e5c394bc5a13
Removing intermediate container e5c394bc5a13
---> c79e2a0a8eca
Step 5/14 : RUN npm install
---> Running in 603d3dc3ccd0
npm WARN deprecated flatten@1.0.3: flatten is deprecated in favor of utility frameworks such as lodash.
npm WARN deprecated urix@0.1.0: Please see https://github.com/lydell/urix#deprecated
npm WARN deprecated source-map-url@0.4.1: See https://github.com/lydell/source-map-url#deprecated
npm WARN deprecated resolve-url@0.2.1: https://github.com/lydell/resolve-url#deprecated
npm WARN deprecated source-map-resolve@0.5.3: See https://github.com/lydell/source-map-resolve#deprecated
npm WARN deprecated chokidar@2.1.8: Chokidar 2 does not receive security updates since 2019. Upgrade to chokidar 3 with 15x fewer dependencies
npm WARN deprecated mkdirp@0.5.1: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)

> @grpc/grpc-js@1.8.16 prepare
> npm run generate-types && npm run compile


> @grpc/grpc-js@1.8.16 generate-types
> proto-loader-gen-types --keepCase --longs String --enums String --defaults --oneofs --includeComments --includeDirs proto/ --include-dirs test/fixtures/ -O src/generated/ --grpcLib ../index channelz.proto


> @grpc/grpc-js@1.8.16 compile
> tsc -p .


added 810 packages, and audited 811 packages in 33s

83 packages are looking for funding
run `npm fund` for details

22 vulnerabilities (11 moderate, 8 high, 3 critical)

To address issues that do not require attention, run:
npm audit fix

To address all issues (including breaking changes), run:
npm audit fix --force

Run `npm audit` for details.
npm notice
npm notice New minor version of npm available! 9.5.1 -> 9.7.2
npm notice Changelog:
npm notice Run `npm install -g npm@9.7.2` to update!
npm notice
Removing intermediate container 603d3dc3ccd0
---> 464ef633aae2
Step 6/14 : WORKDIR /node/src/grpc-node/packages/grpc-js-xds
---> Running in 05134ce7602a
Removing intermediate container 05134ce7602a
---> 07ee83792152
Step 7/14 : RUN npm install
---> Running in 4c0021bf3ede

> @grpc/grpc-js-xds@1.8.2 prepare
> npm run compile


> @grpc/grpc-js-xds@1.8.2 compile
> tsc


added 383 packages, and audited 385 packages in 36s

49 packages are looking for funding
run `npm fund` for details

20 moderate severity vulnerabilities

To address issues that do not require attention, run:
npm audit fix

To address all issues (including breaking changes), run:
npm audit fix --force

Run `npm audit` for details.
Removing intermediate container 4c0021bf3ede
---> 2cc7bb0d7d0a
Step 8/14 : FROM gcr.io/distroless/nodejs18-debian11:latest
latest: Pulling from distroless/nodejs18-debian11
a7ca0d9ba68f: Pulling fs layer
fe5ca62666f0: Pulling fs layer
b02a7525f878: Pulling fs layer
fcb6f6d2c998: Pulling fs layer
e8c73c638ae9: Pulling fs layer
1e3d9b7d1452: Pulling fs layer
4aa0ea1413d3: Pulling fs layer
7c881f9ab25e: Pulling fs layer
5627a970d25e: Pulling fs layer
96266735468f: Pulling fs layer
2758d0c31c8c: Pulling fs layer
08553ba93cfe: Pulling fs layer
dfc02eb7708f: Pulling fs layer
52907d314ddc: Pulling fs layer
4eec690774a4: Pulling fs layer
a22eaaf37419: Pulling fs layer
52907d314ddc: Waiting
4eec690774a4: Waiting
a22eaaf37419: Waiting
1e3d9b7d1452: Waiting
4aa0ea1413d3: Waiting
fcb6f6d2c998: Waiting
2758d0c31c8c: Waiting
7c881f9ab25e: Waiting
96266735468f: Waiting
5627a970d25e: Waiting
e8c73c638ae9: Waiting
08553ba93cfe: Waiting
dfc02eb7708f: Waiting
a7ca0d9ba68f: Verifying Checksum
a7ca0d9ba68f: Download complete
b02a7525f878: Verifying Checksum
b02a7525f878: Download complete
fcb6f6d2c998: Verifying Checksum
fcb6f6d2c998: Download complete
e8c73c638ae9: Verifying Checksum
e8c73c638ae9: Download complete
1e3d9b7d1452: Verifying Checksum
1e3d9b7d1452: Download complete
a7ca0d9ba68f: Pull complete
4aa0ea1413d3: Verifying Checksum
4aa0ea1413d3: Download complete
7c881f9ab25e: Verifying Checksum
7c881f9ab25e: Download complete
5627a970d25e: Verifying Checksum
5627a970d25e: Download complete
fe5ca62666f0: Pull complete
08553ba93cfe: Download complete
2758d0c31c8c: Verifying Checksum
2758d0c31c8c: Download complete
96266735468f: Verifying Checksum
96266735468f: Download complete
dfc02eb7708f: Verifying Checksum
dfc02eb7708f: Download complete
52907d314ddc: Verifying Checksum
52907d314ddc: Download complete
4eec690774a4: Verifying Checksum
4eec690774a4: Download complete
b02a7525f878: Pull complete
fcb6f6d2c998: Pull complete
e8c73c638ae9: Pull complete
1e3d9b7d1452: Pull complete
4aa0ea1413d3: Pull complete
7c881f9ab25e: Pull complete
5627a970d25e: Pull complete
a22eaaf37419: Verifying Checksum
96266735468f: Pull complete
2758d0c31c8c: Pull complete
08553ba93cfe: Pull complete
dfc02eb7708f: Pull complete
52907d314ddc: Pull complete
4eec690774a4: Pull complete
a22eaaf37419: Pull complete
Digest: sha256:c73d0509171d06c166a5e4ffb0b6a547623e5666f2764651f7dff7a3a97f2678
Status: Downloaded newer image for gcr.io/distroless/nodejs18-debian11:latest
---> 93ac4357c47c
Step 9/14 : WORKDIR /node/src/grpc-node
---> Running in ba36788a5b04
Removing intermediate container ba36788a5b04
---> ced8d7ddeb88
Step 10/14 : COPY --from=build /node/src/grpc-node/packages/grpc-js ./packages/grpc-js/
---> 3a6579ceb22b
Step 11/14 : COPY --from=build /node/src/grpc-node/packages/grpc-js-xds ./packages/grpc-js-xds/
---> 7463bf5e637a
Step 12/14 : ENV GRPC_VERBOSITY="DEBUG"
---> Running in 478eff80582d
Removing intermediate container 478eff80582d
---> 2c84ab31fd6b
Step 13/14 : ENV GRPC_TRACE=xds_client,xds_resolver,cds_balancer,eds_balancer,priority,weighted_target,round_robin,resolving_load_balancer,subchannel,keepalive,dns_resolver,fault_injection,http_filter,csds,outlier_detection
---> Running in 6844d6c71c5a
Removing intermediate container 6844d6c71c5a
---> 424b2af290a7
Step 14/14 : ENTRYPOINT [ "node", "/node/src/grpc-node/packages/grpc-js-xds/build/interop/xds-interop-client" ]
---> Running in 590c4bb3e50b
Removing intermediate container 590c4bb3e50b
---> f2951af1ea17
Successfully built f2951af1ea17
Successfully tagged gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
+ gcloud -q auth configure-docker
Adding credentials for all GCR repositories.
WARNING: A long list of credential helpers may cause delays running 'docker build'. We recommend passing the registry name to configure only the registry you are using.
Docker configuration file updated.
+ docker push gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
The push refers to repository [gcr.io/grpc-testing/xds-interop/node-client]
994bbc642553: Preparing
bb3b0096ab92: Preparing
bf67b8c30e20: Preparing
c1840328978b: Preparing
5bf213caca44: Preparing
dcecd3d0367d: Preparing
bb0331ba4692: Preparing
6a1069d9378c: Preparing
1c47a89b8f41: Preparing
c60b28d3f33c: Preparing
4cb10dd2545b: Preparing
d2d7ec0f6756: Preparing
1a73b54f556b: Preparing
e624a5370eca: Preparing
d52f02c6501c: Preparing
ff5700ec5418: Preparing
7bea6b893187: Preparing
6fbdf253bbc2: Preparing
e023e0e48e6e: Preparing
dcecd3d0367d: Waiting
d2d7ec0f6756: Waiting
1a73b54f556b: Waiting
bb0331ba4692: Waiting
e624a5370eca: Waiting
6a1069d9378c: Waiting
7bea6b893187: Waiting
1c47a89b8f41: Waiting
6fbdf253bbc2: Waiting
d52f02c6501c: Waiting
e023e0e48e6e: Waiting
ff5700ec5418: Waiting
c60b28d3f33c: Waiting
4cb10dd2545b: Waiting
c1840328978b: Layer already exists
5bf213caca44: Layer already exists
dcecd3d0367d: Layer already exists
bb0331ba4692: Layer already exists
6a1069d9378c: Layer already exists
1c47a89b8f41: Layer already exists
c60b28d3f33c: Layer already exists
4cb10dd2545b: Layer already exists
d2d7ec0f6756: Layer already exists
1a73b54f556b: Layer already exists
e624a5370eca: Layer already exists
d52f02c6501c: Layer already exists
ff5700ec5418: Layer already exists
6fbdf253bbc2: Layer already exists
7bea6b893187: Layer already exists
e023e0e48e6e: Layer already exists
bf67b8c30e20: Pushed
994bbc642553: Pushed
bb3b0096ab92: Pushed
9441de78f655ada34ada0dc1a8057122eb21f229: digest: sha256:91fee3eb421b8c096e17fea4b2ba7150da0f51e5bd87600813732d763145646f size: 4295
+ is_version_branch dev-v1.8.x
+ '[' 1 -eq 0 ']'
+ [[ dev-v1.8.x == \m\a\s\t\e\r ]]
+ local 'version_regex=^v[0-9]+\.[0-9]+\.x$'
+ [[ dev-v1.8.x =~ ^v[0-9]+\.[0-9]+\.x$ ]]
+ popd
/tmpfs/tmp/tmp.OvLKtVktid/grpc
+ cd /tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver
+ local failed_tests=0
+ test_suites=("baseline_test" "api_listener_test" "change_backend_service_test" "failover_test" "remove_neg_test" "round_robin_test" "outlier_detection_test")
+ for test in '"${test_suites[@]}"'
+ run_test baseline_test
+ local test_name=baseline_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test'
+ python3 -m tests.baseline_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 00:06:35.409363 139975191660288 xds_k8s_testcase.py:130] ----- Testing BaselineTest -----
I0623 00:06:35.453349 139975191660288 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 00:06:35.453752 139975191660288 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 00:06:35.476877 139975191660288 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 00:06:35.498924 139975191660288 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] BaselineTest.test_traffic_director_grpc_setup
I0623 00:06:35.504801 139975191660288 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0706-k30u7
I0623 00:06:35.814898 139975191660288 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/baseline_test
I0623 00:06:36.025840 139975191660288 xds_k8s_testcase.py:625] Found unused xds port: 13078
I0623 00:06:36.026347 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0623 00:06:36.026520 139975191660288 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0706-k30u7"
I0623 00:06:36.030102 139975191660288 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0706-k30u7
type: GRPC
...

I0623 00:06:36.664471 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687503996057-5fec6a43df802-541033d9-b6997b4c
I0623 00:06:38.975141 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.0_create_health_check ---
I0623 00:06:38.975411 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0623 00:06:38.975522 139975191660288 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0706-k30u7"
I0623 00:06:38.982924 139975191660288 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0706-k30u7
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0706-k30u7
protocol: GRPC
...

I0623 00:06:39.661226 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687503999015-5fec6a46b18e4-87519182-ceae4906
I0623 00:07:03.863568 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.1_create_backend_service ---
I0623 00:07:03.863831 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0623 00:07:03.863935 139975191660288 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0706-k30u7": psm-grpc-server:13078 -> psm-interop-backend-service-20230623-0706-k30u7
I0623 00:07:03.875604 139975191660288 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0706-k30u7
hostRules:
- hosts:
- psm-grpc-server:13078
pathMatcher: psm-interop-path-matcher-20230623-0706-k30u7
name: psm-interop-url-map-20230623-0706-k30u7
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0706-k30u7
name: psm-interop-path-matcher-20230623-0706-k30u7
...

I0623 00:07:04.710743 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504023902-5fec6a5e6d98c-75be42a9-0e9f493f
I0623 00:07:07.031355 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.2_create_url_map ---
I0623 00:07:07.031606 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0623 00:07:07.031739 139975191660288 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0706-k30u7 proxy "GRPC" to URL map psm-interop-url-map-20230623-0706-k30u7
I0623 00:07:07.033840 139975191660288 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0706-k30u7
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0706-k30u7
validate_for_proxyless: true
...

I0623 00:07:07.516001 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504027062-5fec6a6170fa1-c986ce5a-4445349e
I0623 00:07:09.897942 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.3_create_target_proxy ---
I0623 00:07:09.898199 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0623 00:07:09.898381 139975191660288 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0706-k30u7" in network "default-vpc": 0.0.0.0:13078 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0706-k30u7
I0623 00:07:09.901162 139975191660288 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0706-k30u7
network: global/networks/default-vpc
portRange: 13078
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0706-k30u7
...

I0623 00:07:10.782384 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504029932-5fec6a642daf4-3023369a-a7a7cf33
I0623 00:07:26.298305 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.4_create_forwarding_rule ---
I0623 00:07:26.298572 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0623 00:07:26.298734 139975191660288 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0706-k30u7: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:07:26.299057 139975191660288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0706-k30u7%22;timeRange=2023-06-23T07:07:26.298839Z%2F2023-06-23T07:37:26.298839Z?project=grpc-testing
I0623 00:07:26.305014 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0706-k30u7
labels:
name: psm-interop-server-20230623-0706-k30u7
owner: xds-k8s-interop-test
...

I0623 00:07:26.391653 139975191660288 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0706-k30u7 created
I0623 00:07:26.395175 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0706-k30u7
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0706-k30u7-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:07:26.459124 139975191660288 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 00:07:26.459406 139975191660288 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 00:07:26.510136 139975191660288 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0706-k30u7-psm-grpc-server in zones=['us-central1-a']
I0623 00:07:26.510424 139975191660288 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0706-k30u7/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:07:26.903391 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0706-k30u7
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:07:26.946410 139975191660288 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 00:07:26.955202 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0706-k30u7
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n3vhe5h0
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n3vhe5h0
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-n3vhe5h0
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0706-k30u7-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:07:27.000324 139975191660288 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 00:07:27.000586 139975191660288 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0623 00:07:28.065598 139975191660288 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-5d8db4b75f-jztjp']
I0623 00:07:28.065928 139975191660288 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5d8db4b75f-jztjp to start
I0623 00:07:30.149252 139975191660288 k8s_base_runner.py:554] Pod psm-grpc-server-5d8db4b75f-jztjp ready, IP: 10.120.4.78
I0623 00:07:30.149571 139975191660288 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5d8db4b75f-jztjp to baseline_test/test_app_logs/psm-interop-server-20230623-0706-k30u7_psm-grpc-server-5d8db4b75f-jztjp.log
I0623 00:07:30.150102 139975001630464 k8s_log_collector.py:70] Starting log collection thread 139975001630464 for psm-grpc-server-5d8db4b75f-jztjp
I0623 00:07:30.150351 139975191660288 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0623 00:07:36.297303 139975191660288 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0623 00:07:36.297800 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.5_start_test_server ---
I0623 00:07:36.297929 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0623 00:07:36.304526 139975191660288 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:07:36.386656 139975191660288 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0706-k30u7-psm-grpc-server" in zone us-central1-a
I0623 00:07:36.386955 139975191660288 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0706-k30u7: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0706-k30u7-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0706-k30u7-psm-grpc-server', zone='us-central1-a')}
I0623 00:07:36.391288 139975191660288 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0706-k30u7-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 00:07:37.374922 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504056418-5fec6a7d700ee-f23097b9-1b5e0ff6
I0623 00:08:27.195031 139975191660288 compute.py:433] Backend psm-interop-server-20230623-0706-k30u7-psm-grpc-server in zone us-central1-a reported healthy
I0623 00:08:31.200844 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.6_add_server_backends_to_backend_service ---
I0623 00:08:31.201128 139975191660288 xds_k8s_testcase.py:239] --- Starting subTest __main__.BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0623 00:08:31.201265 139975191660288 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0706-k30u7: server_target=xds:///psm-grpc-server:13078 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 00:08:31.201521 139975191660288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0706-k30u7%22;timeRange=2023-06-23T07:08:31.201355Z%2F2023-06-23T07:38:31.201355Z?project=grpc-testing
I0623 00:08:31.203844 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0706-k30u7
labels:
name: psm-interop-client-20230623-0706-k30u7
owner: xds-k8s-interop-test
...

I0623 00:08:31.231267 139975191660288 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0706-k30u7 created
I0623 00:08:31.231539 139975191660288 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0706-k30u7/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:08:31.606698 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0706-k30u7
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:08:31.641228 139975191660288 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 00:08:31.651329 139975191660288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0706-k30u7
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-30x19wou
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-30x19wou
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-30x19wou
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:13078"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:08:31.700850 139975191660288 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 00:08:31.701125 139975191660288 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 00:08:32.758998 139975191660288 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5db8457d6-hqp8m']
I0623 00:08:32.759315 139975191660288 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-5db8457d6-hqp8m to start
I0623 00:08:48.026094 139975191660288 k8s_base_runner.py:554] Pod psm-grpc-client-5db8457d6-hqp8m ready, IP: 10.120.5.203
I0623 00:08:48.026445 139975191660288 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-5db8457d6-hqp8m to baseline_test/test_app_logs/psm-interop-client-20230623-0706-k30u7_psm-grpc-client-5db8457d6-hqp8m.log
I0623 00:08:48.026884 139974992975616 k8s_log_collector.py:70] Starting log collection thread 139974992975616 for psm-grpc-client-5db8457d6-hqp8m
I0623 00:08:48.027043 139975191660288 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 00:13:48.856014 139975191660288 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 07:08:31+00:00
last_update_time: 2023-06-23 07:08:31+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 07:08:31+00:00
last_update_time: 2023-06-23 07:08:31+00:00
message: ReplicaSet "psm-grpc-client-5db8457d6" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 00:13:48.856403 139975191660288 xds_k8s_testcase.py:244] --- Finished subTest __main__.BaselineTest.test_traffic_director_grpc_setup.7_start_test_client ---
I0623 00:13:48.856538 139975191660288 xds_k8s_testcase.py:640] ----- TestMethod __main__.BaselineTest.test_traffic_director_grpc_setup teardown -----
I0623 00:13:48.929524 139975191660288 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0706-k30u7"
I0623 00:13:49.463281 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504428970-5fec6be0bb3df-94b7a34a-2ecdbc2f
I0623 00:14:00.472131 139975191660288 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0706-k30u7"
I0623 00:14:00.625912 139975191660288 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:00.626187 139975191660288 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0706-k30u7"
I0623 00:14:01.028307 139975191660288 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:01.028573 139975191660288 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0706-k30u7"
I0623 00:14:01.672863 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504441059-5fec6bec427dd-9fb93140-2c35fc01
I0623 00:14:04.019089 139975191660288 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0706-k30u7"
I0623 00:14:04.564960 139975191660288 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:04.565222 139975191660288 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0706-k30u7"
I0623 00:14:05.075373 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504444597-5fec6befa2530-802ad7d4-875b0411
I0623 00:14:07.477008 139975191660288 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0706-k30u7"
I0623 00:14:07.933107 139975191660288 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:07.933345 139975191660288 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0706-k30u7"
I0623 00:14:08.454046 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504447965-5fec6bf2d8861-60307517-6b06f5b0
I0623 00:14:34.796910 139975191660288 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0706-k30u7"
I0623 00:14:35.236918 139975191660288 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:35.237215 139975191660288 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0706-k30u7"
I0623 00:14:35.638967 139975191660288 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0706-k30u7" not deleted since it does not exist
I0623 00:14:35.639223 139975191660288 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0706-k30u7"
I0623 00:14:36.162591 139975191660288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504475770-5fec6c0d5cfd7-d8f66328-bd23dc43
I0623 00:14:40.751160 139975191660288 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:14:50.818291 139975191660288 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0706-k30u7/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:14:51.175882 139975191660288 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 00:14:51.252989 139975191660288 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0706-k30u7
I0623 00:15:21.367921 139975191660288 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 00:15:31.492265 139975191660288 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 00:15:32.598798 139975191660288 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0706-k30u7/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:15:32.995300 139975191660288 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 00:15:33.065137 139975191660288 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0706-k30u7
I0623 00:17:33.330672 139975191660288 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 00:17:33.331211 139975191660288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-30x19wou:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0706-k30u7%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-30x19wou%22;timeRange=2023-06-23T07:08:31.201345Z%2F2023-06-23T07:15:21.364500Z?project=grpc-testing
I0623 00:17:33.331404 139975191660288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-n3vhe5h0:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0706-k30u7%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-n3vhe5h0%22;timeRange=2023-06-23T07:07:26.298825Z%2F2023-06-23T07:17:33.330457Z?project=grpc-testing
[ FAILED ] BaselineTest.test_traffic_director_grpc_setup
[ FAILED ] BaselineTest.test_traffic_director_grpc_setup
======================================================================
ERROR: test_traffic_director_grpc_setup (__main__.BaselineTest)
BaselineTest.test_traffic_director_grpc_setup
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/baseline_test.py", line 53, in test_traffic_director_grpc_setup
test_client: _XdsTestClient = self.startTestClient(test_server)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_traffic_director_grpc_setup (__main__.BaselineTest)
BaselineTest.test_traffic_director_grpc_setup
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 657.927s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test api_listener_test
+ local test_name=api_listener_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test'
+ python3 -m tests.api_listener_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 00:17:34.450784 139967368124160 xds_k8s_testcase.py:130] ----- Testing ApiListenerTest -----
I0623 00:17:34.451214 139967368124160 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 00:17:34.451520 139967368124160 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 00:17:34.474837 139967368124160 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 00:17:34.497417 139967368124160 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ApiListenerTest.test_api_listener
I0623 00:17:34.503326 139967368124160 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0717-k8b6g
I0623 00:17:34.808012 139967368124160 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/api_listener_test
I0623 00:17:35.043986 139967368124160 xds_k8s_testcase.py:625] Found unused xds port: 8984
I0623 00:17:35.044430 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0623 00:17:35.044567 139967368124160 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0717-k8b6g"
I0623 00:17:35.048214 139967368124160 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0717-k8b6g
type: GRPC
...

I0623 00:17:35.729722 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504655084-5fec6cb85e9f3-47b93a10-39551ce4
I0623 00:17:38.053548 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.00_create_health_check ---
I0623 00:17:38.053793 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0623 00:17:38.053896 139967368124160 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0717-k8b6g"
I0623 00:17:38.061499 139967368124160 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0717-k8b6g
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0717-k8b6g
protocol: GRPC
...

I0623 00:17:38.785279 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504658097-5fec6cbb3e597-efdb1115-63cef931
I0623 00:18:20.088240 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.01_create_backend_services ---
I0623 00:18:20.088535 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0623 00:18:20.088643 139967368124160 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0717-k8b6g": psm-grpc-server:8984 -> psm-interop-backend-service-20230623-0717-k8b6g
I0623 00:18:20.100236 139967368124160 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0717-k8b6g
hostRules:
- hosts:
- psm-grpc-server:8984
pathMatcher: psm-interop-path-matcher-20230623-0717-k8b6g
name: psm-interop-url-map-20230623-0717-k8b6g
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0717-k8b6g
name: psm-interop-path-matcher-20230623-0717-k8b6g
...

I0623 00:18:20.744892 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504700137-5fec6ce356079-dbd67d03-ce479668
I0623 00:18:23.093208 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.02_create_default_url_map ---
I0623 00:18:23.093458 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0623 00:18:23.093585 139967368124160 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0717-k8b6g proxy "GRPC" to URL map psm-interop-url-map-20230623-0717-k8b6g
I0623 00:18:23.095706 139967368124160 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0717-k8b6g
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0717-k8b6g
validate_for_proxyless: true
...

I0623 00:18:23.765015 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504703128-5fec6ce630263-66bc3fa9-e373e4a0
I0623 00:18:26.132652 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.03_create_default_target_proxy ---
I0623 00:18:26.132905 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0623 00:18:26.133016 139967368124160 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0717-k8b6g" in network "default-vpc": 0.0.0.0:8984 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0717-k8b6g
I0623 00:18:26.136070 139967368124160 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0717-k8b6g
network: global/networks/default-vpc
portRange: 8984
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0717-k8b6g
...

I0623 00:18:27.024251 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504706173-5fec6ce917901-59b2eb55-b235daf4
I0623 00:18:40.134114 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.04_create_default_forwarding_rule ---
I0623 00:18:40.134402 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0623 00:18:40.134514 139967368124160 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0717-k8b6g: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:18:40.134818 139967368124160 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0717-k8b6g%22;timeRange=2023-06-23T07:18:40.134627Z%2F2023-06-23T07:48:40.134627Z?project=grpc-testing
I0623 00:18:40.140793 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0717-k8b6g
labels:
name: psm-interop-server-20230623-0717-k8b6g
owner: xds-k8s-interop-test
...

I0623 00:18:40.240877 139967368124160 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0717-k8b6g created
I0623 00:18:40.244434 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0717-k8b6g
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0717-k8b6g-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:18:40.291376 139967368124160 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 00:18:40.291666 139967368124160 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 00:18:40.333589 139967368124160 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0717-k8b6g-psm-grpc-server in zones=['us-central1-a']
I0623 00:18:40.333847 139967368124160 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0717-k8b6g/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:18:40.815894 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0717-k8b6g
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:18:40.848986 139967368124160 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 00:18:40.857907 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0717-k8b6g
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-pvp7fanx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-pvp7fanx
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-pvp7fanx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0717-k8b6g-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:18:40.904341 139967368124160 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 00:18:40.904608 139967368124160 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0623 00:18:41.000424 139967368124160 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-6848f6fffb-sf8db']
I0623 00:18:41.000723 139967368124160 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-6848f6fffb-sf8db to start
I0623 00:18:44.102199 139967368124160 k8s_base_runner.py:554] Pod psm-grpc-server-6848f6fffb-sf8db ready, IP: 10.120.2.181
I0623 00:18:44.102544 139967368124160 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-6848f6fffb-sf8db to api_listener_test/test_app_logs/psm-interop-server-20230623-0717-k8b6g_psm-grpc-server-6848f6fffb-sf8db.log
I0623 00:18:44.103067 139967178094336 k8s_log_collector.py:70] Starting log collection thread 139967178094336 for psm-grpc-server-6848f6fffb-sf8db
I0623 00:18:44.103293 139967368124160 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0623 00:18:52.259360 139967368124160 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0623 00:18:52.259832 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.05_start_test_server ---
I0623 00:18:52.259953 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0623 00:18:52.266620 139967368124160 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:18:52.344707 139967368124160 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0717-k8b6g-psm-grpc-server" in zone us-central1-a
I0623 00:18:52.344928 139967368124160 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0717-k8b6g: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0717-k8b6g-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0717-k8b6g-psm-grpc-server', zone='us-central1-a')}
I0623 00:18:52.349649 139967368124160 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0717-k8b6g-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 00:18:53.328684 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687504732382-5fec6d0216678-51a7eb67-ad40a4b9
I0623 00:20:02.814908 139967368124160 compute.py:433] Backend psm-interop-server-20230623-0717-k8b6g-psm-grpc-server in zone us-central1-a reported healthy
I0623 00:20:06.820803 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.06_add_server_backends_to_backend_services ---
I0623 00:20:06.821073 139967368124160 xds_k8s_testcase.py:239] --- Starting subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0623 00:20:06.821202 139967368124160 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0717-k8b6g: server_target=xds:///psm-grpc-server:8984 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 00:20:06.821449 139967368124160 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0717-k8b6g%22;timeRange=2023-06-23T07:20:06.821288Z%2F2023-06-23T07:50:06.821288Z?project=grpc-testing
I0623 00:20:06.823860 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0717-k8b6g
labels:
name: psm-interop-client-20230623-0717-k8b6g
owner: xds-k8s-interop-test
...

I0623 00:20:06.844755 139967368124160 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0717-k8b6g created
I0623 00:20:06.845022 139967368124160 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0717-k8b6g/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:20:07.319288 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0717-k8b6g
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:20:07.359185 139967368124160 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 00:20:07.369123 139967368124160 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0717-k8b6g
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8qr2ekq9
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8qr2ekq9
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-8qr2ekq9
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:8984"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:20:07.414161 139967368124160 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 00:20:07.414494 139967368124160 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 00:20:07.476694 139967368124160 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-64565c6cb9-wwlhv']
I0623 00:20:07.476992 139967368124160 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-64565c6cb9-wwlhv to start
I0623 00:20:23.779413 139967368124160 k8s_base_runner.py:554] Pod psm-grpc-client-64565c6cb9-wwlhv ready, IP: 10.120.0.250
I0623 00:20:23.779799 139967368124160 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-64565c6cb9-wwlhv to api_listener_test/test_app_logs/psm-interop-client-20230623-0717-k8b6g_psm-grpc-client-64565c6cb9-wwlhv.log
I0623 00:20:23.780393 139967169439488 k8s_log_collector.py:70] Starting log collection thread 139967169439488 for psm-grpc-client-64565c6cb9-wwlhv
I0623 00:20:23.780723 139967368124160 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 00:25:24.499941 139967368124160 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 07:20:07+00:00
last_update_time: 2023-06-23 07:20:07+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 07:20:07+00:00
last_update_time: 2023-06-23 07:20:07+00:00
message: ReplicaSet "psm-grpc-client-64565c6cb9" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 00:25:24.500408 139967368124160 xds_k8s_testcase.py:244] --- Finished subTest __main__.ApiListenerTest.test_api_listener.07_start_test_client ---
I0623 00:25:24.500560 139967368124160 xds_k8s_testcase.py:640] ----- TestMethod __main__.ApiListenerTest.test_api_listener teardown -----
I0623 00:25:24.577235 139967368124160 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0717-k8b6g"
I0623 00:25:25.272828 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505124631-5fec6e782a1f0-c03749bb-9d157c01
I0623 00:25:42.804668 139967368124160 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0717-k8b6g"
I0623 00:25:42.943397 139967368124160 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:25:42.943665 139967368124160 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0717-k8b6g"
I0623 00:25:43.329910 139967368124160 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:25:43.330163 139967368124160 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0717-k8b6g"
I0623 00:25:44.002313 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505143376-5fec6e8a0a992-438031d8-dc4c6ad6
I0623 00:25:46.317431 139967368124160 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0717-k8b6g"
I0623 00:25:46.787176 139967368124160 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:25:46.787436 139967368124160 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0717-k8b6g"
I0623 00:25:47.498702 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505146835-5fec6e8d57206-2bc9cc38-06a2a6ee
I0623 00:25:49.839470 139967368124160 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0717-k8b6g"
I0623 00:25:50.230608 139967368124160 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:25:50.230855 139967368124160 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0717-k8b6g"
I0623 00:25:50.867068 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505150278-5fec6e909fbc5-5cd30189-bc8d87d5
I0623 00:26:17.186995 139967368124160 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0717-k8b6g"
I0623 00:26:17.625534 139967368124160 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:26:17.625853 139967368124160 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0717-k8b6g"
I0623 00:26:17.979895 139967368124160 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0717-k8b6g" not deleted since it does not exist
I0623 00:26:17.980146 139967368124160 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0717-k8b6g"
I0623 00:26:18.561463 139967368124160 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505178102-5fec6eab28989-28974e04-175b471e
I0623 00:26:23.061595 139967368124160 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:26:33.129748 139967368124160 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0717-k8b6g/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:26:33.568103 139967368124160 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 00:26:33.645739 139967368124160 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0717-k8b6g
I0623 00:27:03.796349 139967368124160 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 00:27:13.907709 139967368124160 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 00:27:14.986027 139967368124160 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0717-k8b6g/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:27:15.378146 139967368124160 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 00:27:15.447378 139967368124160 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0717-k8b6g
I0623 00:28:45.639423 139967368124160 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 00:28:45.639886 139967368124160 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-8qr2ekq9:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0717-k8b6g%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-8qr2ekq9%22;timeRange=2023-06-23T07:20:06.821278Z%2F2023-06-23T07:27:03.792403Z?project=grpc-testing
I0623 00:28:45.640065 139967368124160 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-pvp7fanx:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0717-k8b6g%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-pvp7fanx%22;timeRange=2023-06-23T07:18:40.134614Z%2F2023-06-23T07:28:45.639213Z?project=grpc-testing
[ FAILED ] ApiListenerTest.test_api_listener
[ FAILED ] ApiListenerTest.test_api_listener
======================================================================
ERROR: test_api_listener (__main__.ApiListenerTest)
ApiListenerTest.test_api_listener
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/api_listener_test.py", line 70, in test_api_listener
test_client = self.startTestClient(test_server)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_api_listener (__main__.ApiListenerTest)
ApiListenerTest.test_api_listener
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 671.194s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test change_backend_service_test
+ local test_name=change_backend_service_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test'
+ python3 -m tests.change_backend_service_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 00:28:46.827989 140112310892288 xds_k8s_testcase.py:130] ----- Testing ChangeBackendServiceTest -----
I0623 00:28:46.828484 140112310892288 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 00:28:46.828808 140112310892288 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 00:28:46.851541 140112310892288 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 00:28:46.874664 140112310892288 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] ChangeBackendServiceTest.test_change_backend_service
I0623 00:28:46.880648 140112310892288 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0728-kasbi
I0623 00:28:47.230202 140112310892288 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/change_backend_service_test
I0623 00:28:47.484694 140112310892288 xds_k8s_testcase.py:625] Found unused xds port: 10624
I0623 00:28:47.487280 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0623 00:28:47.487545 140112310892288 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0728-kasbi"
I0623 00:28:47.491215 140112310892288 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0728-kasbi
type: GRPC
...

I0623 00:28:48.110384 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505327539-5fec6f39ac610-88044e26-2256f6cf
I0623 00:28:50.452420 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.00_create_health_check ---
I0623 00:28:50.452662 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0623 00:28:50.452789 140112310892288 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0728-kasbi"
I0623 00:28:50.460820 140112310892288 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0728-kasbi
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0728-kasbi
protocol: GRPC
...

I0623 00:28:51.036643 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505330509-5fec6f3c815d0-a3886193-22829ebd
I0623 00:29:54.112428 140112310892288 traffic_director.py:300] Creating GRPC Alternative Backend Service "psm-interop-backend-service-alt-20230623-0728-kasbi"
I0623 00:29:54.117547 140112310892288 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0728-kasbi
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-alt-20230623-0728-kasbi
protocol: GRPC
...

I0623 00:29:54.892329 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505394163-5fec6f7935f2e-40e5aaa1-d880f4af
I0623 00:30:18.771349 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.01_create_backend_services ---
I0623 00:30:18.771610 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0623 00:30:18.771715 140112310892288 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0728-kasbi": psm-grpc-server:10624 -> psm-interop-backend-service-20230623-0728-kasbi
I0623 00:30:18.784049 140112310892288 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0728-kasbi
hostRules:
- hosts:
- psm-grpc-server:10624
pathMatcher: psm-interop-path-matcher-20230623-0728-kasbi
name: psm-interop-url-map-20230623-0728-kasbi
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0728-kasbi
name: psm-interop-path-matcher-20230623-0728-kasbi
...

I0623 00:30:19.457796 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505418830-5fec6f90bc2be-3daa8591-afa490c9
I0623 00:30:21.793404 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.02_create_url_map ---
I0623 00:30:21.793702 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0623 00:30:21.793821 140112310892288 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0728-kasbi proxy "GRPC" to URL map psm-interop-url-map-20230623-0728-kasbi
I0623 00:30:21.796205 140112310892288 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0728-kasbi
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0728-kasbi
validate_for_proxyless: true
...

I0623 00:30:22.456139 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505421842-5fec6f939b8d7-e2f252bd-0ba1e1c2
I0623 00:30:24.765510 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.03_create_target_proxy ---
I0623 00:30:24.765818 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0623 00:30:24.765980 140112310892288 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0728-kasbi" in network "default-vpc": 0.0.0.0:10624 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0728-kasbi
I0623 00:30:24.769118 140112310892288 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0728-kasbi
network: global/networks/default-vpc
portRange: 10624
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0728-kasbi
...

I0623 00:30:25.367812 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505424815-5fec6f9671620-1c177cb6-dfd38267
I0623 00:31:10.747322 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.04_create_forwarding_rule ---
I0623 00:31:10.747581 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0623 00:31:10.747711 140112310892288 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0728-kasbi: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:31:10.748010 140112310892288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0728-kasbi%22;timeRange=2023-06-23T07:31:10.747808Z%2F2023-06-23T08:01:10.747808Z?project=grpc-testing
I0623 00:31:10.754356 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0728-kasbi
labels:
name: psm-interop-server-20230623-0728-kasbi
owner: xds-k8s-interop-test
...

I0623 00:31:10.821507 140112310892288 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0728-kasbi created
I0623 00:31:10.825082 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0728-kasbi
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0728-kasbi-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:31:10.876240 140112310892288 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 00:31:10.876500 140112310892288 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 00:31:10.917299 140112310892288 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0728-kasbi-psm-grpc-server in zones=['us-central1-a']
I0623 00:31:10.917540 140112310892288 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0728-kasbi/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:31:11.434832 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0728-kasbi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:31:11.470339 140112310892288 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 00:31:11.479509 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0728-kasbi
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qxctj1td
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qxctj1td
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-qxctj1td
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0728-kasbi-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:31:11.536292 140112310892288 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 00:31:11.536593 140112310892288 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0623 00:31:11.662757 140112310892288 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-84888488b9-rv265']
I0623 00:31:11.663057 140112310892288 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-84888488b9-rv265 to start
I0623 00:31:14.769716 140112310892288 k8s_base_runner.py:554] Pod psm-grpc-server-84888488b9-rv265 ready, IP: 10.120.0.254
I0623 00:31:14.770066 140112310892288 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-84888488b9-rv265 to change_backend_service_test/test_app_logs/psm-interop-server-20230623-0728-kasbi_psm-grpc-server-84888488b9-rv265.log
I0623 00:31:14.770642 140112120862464 k8s_log_collector.py:70] Starting log collection thread 140112120862464 for psm-grpc-server-84888488b9-rv265
I0623 00:31:14.770848 140112310892288 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0623 00:31:21.899038 140112310892288 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0623 00:31:21.899500 140112310892288 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230623-0728-kasbi: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:31:21.899755 140112310892288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0728-kasbi%22;timeRange=2023-06-23T07:31:21.899617Z%2F2023-06-23T08:01:21.899617Z?project=grpc-testing
I0623 00:31:21.936376 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0728-kasbi
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:31:21.953275 140112310892288 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0623 00:31:21.953534 140112310892288 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0623 00:31:21.989776 140112310892288 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt in zones=['us-central1-a']
I0623 00:31:21.990053 140112310892288 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0728-kasbi/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:31:23.868373 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0728-kasbi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:31:23.878506 140112310892288 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0623 00:31:23.887420 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0728-kasbi
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-nbd7n2p8
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-nbd7n2p8
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-nbd7n2p8
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:31:23.909729 140112310892288 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0623 00:31:23.910007 140112310892288 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0623 00:31:24.947075 140112310892288 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7876c67fbc-5lx2h']
I0623 00:31:24.947350 140112310892288 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-7876c67fbc-5lx2h to start
I0623 00:31:27.023376 140112310892288 k8s_base_runner.py:554] Pod psm-grpc-server-alt-7876c67fbc-5lx2h ready, IP: 10.120.1.105
I0623 00:31:27.023783 140112310892288 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-7876c67fbc-5lx2h to change_backend_service_test/test_app_logs/psm-interop-server-20230623-0728-kasbi_psm-grpc-server-alt-7876c67fbc-5lx2h.log
I0623 00:31:27.024291 140112112469760 k8s_log_collector.py:70] Starting log collection thread 140112112469760 for psm-grpc-server-alt-7876c67fbc-5lx2h
I0623 00:31:27.024499 140112310892288 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0623 00:31:32.132378 140112310892288 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0623 00:31:32.132960 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.05_start_test_servers ---
I0623 00:31:32.133113 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0623 00:31:32.140147 140112310892288 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:31:32.225903 140112310892288 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0728-kasbi-psm-grpc-server" in zone us-central1-a
I0623 00:31:32.226163 140112310892288 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0728-kasbi: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0728-kasbi-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0728-kasbi-psm-grpc-server', zone='us-central1-a')}
I0623 00:31:32.230748 140112310892288 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0728-kasbi-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 00:31:33.004820 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505492279-5fec6fd6c7f6f-96641f65-746b9daa
I0623 00:32:38.074660 140112310892288 compute.py:433] Backend psm-interop-server-20230623-0728-kasbi-psm-grpc-server in zone us-central1-a reported healthy
I0623 00:32:42.089799 140112310892288 traffic_director.py:328] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:32:42.170109 140112310892288 traffic_director.py:331] Loaded NEG "psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt" in zone us-central1-a
I0623 00:32:42.170424 140112310892288 traffic_director.py:338] Adding backends to Backend Service psm-interop-backend-service-alt-20230623-0728-kasbi: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt', zone='us-central1-a')}
I0623 00:32:42.174706 140112310892288 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0728-kasbi-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0623 00:32:43.083331 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505562224-5fec70197c643-1524d5a2-07fcd9b5
I0623 00:33:37.355742 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.06_add_server_backends_to_backend_services ---
I0623 00:33:37.356091 140112310892288 xds_k8s_testcase.py:239] --- Starting subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0623 00:33:37.356250 140112310892288 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0728-kasbi: server_target=xds:///psm-grpc-server:10624 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 00:33:37.356554 140112310892288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0728-kasbi%22;timeRange=2023-06-23T07:33:37.356348Z%2F2023-06-23T08:03:37.356348Z?project=grpc-testing
I0623 00:33:37.359117 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0728-kasbi
labels:
name: psm-interop-client-20230623-0728-kasbi
owner: xds-k8s-interop-test
...

I0623 00:33:37.418403 140112310892288 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0728-kasbi created
I0623 00:33:37.418665 140112310892288 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0728-kasbi/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:33:37.810290 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0728-kasbi
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:33:37.845232 140112310892288 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 00:33:37.855262 140112310892288 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0728-kasbi
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-93osul04
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-93osul04
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-93osul04
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:10624"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:33:37.915396 140112310892288 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 00:33:37.915678 140112310892288 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 00:33:37.979071 140112310892288 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-7677665475-j2tbp']
I0623 00:33:37.979383 140112310892288 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-7677665475-j2tbp to start
I0623 00:33:53.279050 140112310892288 k8s_base_runner.py:554] Pod psm-grpc-client-7677665475-j2tbp ready, IP: 10.120.4.84
I0623 00:33:53.279370 140112310892288 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-7677665475-j2tbp to change_backend_service_test/test_app_logs/psm-interop-client-20230623-0728-kasbi_psm-grpc-client-7677665475-j2tbp.log
I0623 00:33:53.279871 140112033740544 k8s_log_collector.py:70] Starting log collection thread 140112033740544 for psm-grpc-client-7677665475-j2tbp
I0623 00:33:53.280035 140112310892288 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 00:38:53.852954 140112310892288 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 07:33:38+00:00
last_update_time: 2023-06-23 07:33:38+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 07:33:37+00:00
last_update_time: 2023-06-23 07:33:38+00:00
message: ReplicaSet "psm-grpc-client-7677665475" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 00:38:53.853364 140112310892288 xds_k8s_testcase.py:244] --- Finished subTest __main__.ChangeBackendServiceTest.test_change_backend_service.07_start_test_client ---
I0623 00:38:53.853508 140112310892288 xds_k8s_testcase.py:640] ----- TestMethod __main__.ChangeBackendServiceTest.test_change_backend_service teardown -----
I0623 00:38:53.928748 140112310892288 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0728-kasbi"
I0623 00:38:54.740251 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505933972-5fec717c0338f-c6731017-18480270
I0623 00:39:07.953005 140112310892288 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0728-kasbi"
I0623 00:39:08.078464 140112310892288 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0728-kasbi" not deleted since it does not exist
I0623 00:39:08.078705 140112310892288 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0728-kasbi"
I0623 00:39:08.522423 140112310892288 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0728-kasbi" not deleted since it does not exist
I0623 00:39:08.522672 140112310892288 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0728-kasbi"
I0623 00:39:09.100977 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505948561-5fec7189eceed-ea005b45-a09eb542
I0623 00:39:11.470626 140112310892288 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0728-kasbi"
I0623 00:39:11.791740 140112310892288 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0728-kasbi" not deleted since it does not exist
I0623 00:39:11.791985 140112310892288 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0728-kasbi"
I0623 00:39:12.584939 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505951833-5fec718d0bcb1-b947f239-998ba466
I0623 00:39:17.056966 140112310892288 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0728-kasbi"
I0623 00:39:17.510188 140112310892288 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0728-kasbi" not deleted since it does not exist
I0623 00:39:17.510497 140112310892288 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0728-kasbi"
I0623 00:39:18.200399 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505957550-5fec71927f95c-cc9094c0-14892040
I0623 00:39:52.806021 140112310892288 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0728-kasbi"
I0623 00:39:53.489681 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687505992845-5fec71b42878d-fe034bed-8814a019
I0623 00:40:21.779431 140112310892288 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0728-kasbi"
I0623 00:40:22.246336 140112310892288 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0728-kasbi" not deleted since it does not exist
I0623 00:40:22.246661 140112310892288 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0728-kasbi"
I0623 00:40:22.732595 140112310892288 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506022352-5fec71d04c7aa-dbb8ec4a-413f1014
I0623 00:40:27.281434 140112310892288 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:40:37.369689 140112310892288 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0728-kasbi/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:40:37.872607 140112310892288 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 00:40:37.941397 140112310892288 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0728-kasbi
I0623 00:41:08.039678 140112310892288 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 00:41:18.141792 140112310892288 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 00:41:18.215927 140112310892288 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0728-kasbi/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:41:18.614659 140112310892288 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 00:41:18.674395 140112310892288 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0728-kasbi
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/exceptions.py:91: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
self.headers = http_resp.getheaders()
I0623 00:42:18.836846 140112310892288 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
I0623 00:42:18.865881 140112310892288 k8s_base_runner.py:466] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0623 00:42:18.866208 140112310892288 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
I0623 00:42:18.904329 140112310892288 k8s_base_runner.py:478] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0623 00:42:18.904632 140112310892288 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0728-kasbi/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:42:19.300628 140112310892288 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
I0623 00:42:19.343311 140112310892288 k8s_base_runner.py:491] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0623 00:42:19.343586 140112310892288 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0728-kasbi
I0623 00:42:19.379182 140112310892288 k8s_base_runner.py:503] Namespace psm-interop-server-20230623-0728-kasbi deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230623-0728-kasbi" not found
I0623 00:42:19.379697 140112310892288 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 00:42:19.380003 140112310892288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-93osul04:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0728-kasbi%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-93osul04%22;timeRange=2023-06-23T07:33:37.356337Z%2F2023-06-23T07:41:08.036448Z?project=grpc-testing
I0623 00:42:19.380161 140112310892288 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-qxctj1td:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0728-kasbi%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-qxctj1td%22;timeRange=2023-06-23T07:31:10.747795Z%2F2023-06-23T07:42:18.836475Z?project=grpc-testing
[ FAILED ] ChangeBackendServiceTest.test_change_backend_service
[ FAILED ] ChangeBackendServiceTest.test_change_backend_service
======================================================================
ERROR: test_change_backend_service (__main__.ChangeBackendServiceTest)
ChangeBackendServiceTest.test_change_backend_service
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/change_backend_service_test.py", line 100, in test_change_backend_service
test_client = self.startTestClient(default_test_servers[0])
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_change_backend_service (__main__.ChangeBackendServiceTest)
ChangeBackendServiceTest.test_change_backend_service
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 812.557s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test failover_test
+ local test_name=failover_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test'
+ python3 -m tests.failover_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 00:42:20.506812 139642840954624 xds_k8s_testcase.py:130] ----- Testing FailoverTest -----
I0623 00:42:20.507224 139642840954624 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 00:42:20.507522 139642840954624 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 00:42:20.530329 139642840954624 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 00:42:20.552864 139642840954624 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] FailoverTest.test_failover
I0623 00:42:20.558843 139642840954624 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0742-4tvuu
I0623 00:42:20.975913 139642840954624 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/failover_test
I0623 00:42:21.296021 139642840954624 xds_k8s_testcase.py:625] Found unused xds port: 20929
I0623 00:42:21.298558 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0623 00:42:21.298814 139642840954624 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0742-4tvuu"
I0623 00:42:21.302405 139642840954624 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0742-4tvuu
type: GRPC
...

I0623 00:42:21.814484 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506141333-5fec7241c48c5-71c89596-050bcd4d
I0623 00:42:24.183818 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.00_create_health_check ---
I0623 00:42:24.184081 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0623 00:42:24.184188 139642840954624 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0742-4tvuu"
I0623 00:42:24.191818 139642840954624 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0742-4tvuu
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0742-4tvuu
protocol: GRPC
...

I0623 00:42:24.887129 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506144223-5fec724486099-6df7e011-8e553dce
I0623 00:43:42.826908 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.01_create_backend_services ---
I0623 00:43:42.827238 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0623 00:43:42.827357 139642840954624 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0742-4tvuu": psm-grpc-server:20929 -> psm-interop-backend-service-20230623-0742-4tvuu
I0623 00:43:42.839970 139642840954624 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0742-4tvuu
hostRules:
- hosts:
- psm-grpc-server:20929
pathMatcher: psm-interop-path-matcher-20230623-0742-4tvuu
name: psm-interop-url-map-20230623-0742-4tvuu
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0742-4tvuu
name: psm-interop-path-matcher-20230623-0742-4tvuu
...

I0623 00:43:43.397065 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506222865-5fec728f85ba1-31ee1793-6c25f7bb
I0623 00:43:45.784287 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.02_create_url_map ---
I0623 00:43:45.784527 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0623 00:43:45.784637 139642840954624 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0742-4tvuu proxy "GRPC" to URL map psm-interop-url-map-20230623-0742-4tvuu
I0623 00:43:45.786810 139642840954624 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0742-4tvuu
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0742-4tvuu
validate_for_proxyless: true
...

I0623 00:43:46.539828 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506225815-5fec7292560e7-354f2e2d-aa10e7bf
I0623 00:43:48.835429 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.03_create_target_proxy ---
I0623 00:43:48.835669 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0623 00:43:48.835811 139642840954624 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0742-4tvuu" in network "default-vpc": 0.0.0.0:20929 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0742-4tvuu
I0623 00:43:48.838648 139642840954624 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0742-4tvuu
network: global/networks/default-vpc
portRange: 20929
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0742-4tvuu
...

I0623 00:43:49.742333 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506228868-5fec72953f4d5-0b3cad3f-b89c7e12
I0623 00:44:07.189936 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.04_create_forwarding_rule ---
I0623 00:44:07.190199 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0623 00:44:07.190359 139642840954624 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0742-4tvuu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=3
I0623 00:44:07.190685 139642840954624 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0742-4tvuu%22;timeRange=2023-06-23T07:44:07.190485Z%2F2023-06-23T08:14:07.190485Z?project=grpc-testing
I0623 00:44:07.197059 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0742-4tvuu
labels:
name: psm-interop-server-20230623-0742-4tvuu
owner: xds-k8s-interop-test
...

I0623 00:44:07.267215 139642840954624 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0742-4tvuu created
I0623 00:44:07.270972 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0742-4tvuu-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:44:07.312110 139642840954624 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 00:44:07.312371 139642840954624 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 00:44:07.351513 139642840954624 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0742-4tvuu-psm-grpc-server in zones=['us-central1-a']
I0623 00:44:07.351755 139642840954624 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0742-4tvuu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:44:07.802357 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:44:07.840102 139642840954624 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 00:44:07.849523 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-npf1aae2
owner: xds-k8s-interop-test
spec:
replicas: 3
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-npf1aae2
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-npf1aae2
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0742-4tvuu-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:44:07.902601 139642840954624 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 00:44:07.902880 139642840954624 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 3 pod(s)
I0623 00:44:09.004657 139642840954624 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 3 pod(s): ['psm-grpc-server-5c5d46d88d-2mddt', 'psm-grpc-server-5c5d46d88d-7rfhb', 'psm-grpc-server-5c5d46d88d-7vl6n']
I0623 00:44:09.005232 139642840954624 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5c5d46d88d-2mddt to start
I0623 00:44:11.081383 139642840954624 k8s_base_runner.py:554] Pod psm-grpc-server-5c5d46d88d-2mddt ready, IP: 10.120.4.89
I0623 00:44:11.081731 139642840954624 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5c5d46d88d-2mddt to failover_test/test_app_logs/psm-interop-server-20230623-0742-4tvuu_psm-grpc-server-5c5d46d88d-2mddt.log
I0623 00:44:11.082294 139642650662656 k8s_log_collector.py:70] Starting log collection thread 139642650662656 for psm-grpc-server-5c5d46d88d-2mddt
I0623 00:44:11.082466 139642840954624 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5c5d46d88d-7rfhb to start
I0623 00:44:11.143816 139642840954624 k8s_base_runner.py:554] Pod psm-grpc-server-5c5d46d88d-7rfhb ready, IP: 10.120.7.237
I0623 00:44:11.144160 139642840954624 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5c5d46d88d-7rfhb to failover_test/test_app_logs/psm-interop-server-20230623-0742-4tvuu_psm-grpc-server-5c5d46d88d-7rfhb.log
I0623 00:44:11.144684 139642642269952 k8s_log_collector.py:70] Starting log collection thread 139642642269952 for psm-grpc-server-5c5d46d88d-7rfhb
I0623 00:44:11.144903 139642840954624 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-5c5d46d88d-7vl6n to start
I0623 00:44:11.195856 139642840954624 k8s_base_runner.py:554] Pod psm-grpc-server-5c5d46d88d-7vl6n ready, IP: 10.120.8.86
I0623 00:44:11.196201 139642840954624 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-5c5d46d88d-7vl6n to failover_test/test_app_logs/psm-interop-server-20230623-0742-4tvuu_psm-grpc-server-5c5d46d88d-7vl6n.log
I0623 00:44:11.196681 139642633877248 k8s_log_collector.py:70] Starting log collection thread 139642633877248 for psm-grpc-server-5c5d46d88d-7vl6n
I0623 00:44:11.196898 139642840954624 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 3 available replica(s)
I0623 00:44:19.388976 139642840954624 k8s_base_runner.py:522] Deployment psm-grpc-server has 3 replicas available
I0623 00:44:19.389582 139642840954624 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230623-0742-4tvuu: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:44:19.389857 139642840954624 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0742-4tvuu%22;timeRange=2023-06-23T07:44:19.389730Z%2F2023-06-23T08:14:19.389730Z?project=grpc-testing
I0623 00:44:19.391980 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0742-4tvuu
labels:
name: psm-interop-server-20230623-0742-4tvuu
owner: xds-k8s-interop-test
...

I0623 00:44:19.584355 139642840954624 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0742-4tvuu created
I0623 00:44:19.587960 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:44:19.676295 139642840954624 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0623 00:44:19.676580 139642840954624 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0623 00:44:19.786009 139642840954624 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt in zones=['us-west1-b']
I0623 00:44:19.786368 139642840954624 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0742-4tvuu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:44:20.233067 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:44:20.300724 139642840954624 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0623 00:44:20.309177 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0742-4tvuu
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-460j5ksb
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-460j5ksb
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-460j5ksb
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:44:20.389374 139642840954624 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0623 00:44:20.389659 139642840954624 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0623 00:44:20.539513 139642840954624 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-9c5697d87-zwnmc']
I0623 00:44:20.539794 139642840954624 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-9c5697d87-zwnmc to start
I0623 00:44:24.851175 139642840954624 k8s_base_runner.py:554] Pod psm-grpc-server-alt-9c5697d87-zwnmc ready, IP: 10.108.1.155
I0623 00:44:24.851590 139642840954624 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-9c5697d87-zwnmc to failover_test/test_app_logs/psm-interop-server-20230623-0742-4tvuu_psm-grpc-server-alt-9c5697d87-zwnmc.log
I0623 00:44:24.852093 139642625484544 k8s_log_collector.py:70] Starting log collection thread 139642625484544 for psm-grpc-server-alt-9c5697d87-zwnmc
I0623 00:44:24.852352 139642840954624 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0623 00:44:32.280004 139642840954624 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0623 00:44:32.280467 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.05_start_test_servers ---
I0623 00:44:32.280596 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0623 00:44:32.291792 139642840954624 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:44:32.384612 139642840954624 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0742-4tvuu-psm-grpc-server" in zone us-central1-a
I0623 00:44:32.384858 139642840954624 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0742-4tvuu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0742-4tvuu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server', zone='us-central1-a')}
I0623 00:44:32.389100 139642840954624 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server
maxRatePerEndpoint: 100
...

I0623 00:44:33.747709 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506272413-5fec72bec680e-f197e649-86d2f754
I0623 00:45:30.131132 139642840954624 compute.py:433] Backend psm-interop-server-20230623-0742-4tvuu-psm-grpc-server in zone us-central1-a reported healthy
I0623 00:45:34.180674 139642840954624 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:45:34.288515 139642840954624 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt" in zone us-west1-b
I0623 00:45:34.288775 139642840954624 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0742-4tvuu: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt', zone='us-west1-b'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0742-4tvuu-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server', zone='us-central1-a')}
I0623 00:45:34.293990 139642840954624 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-west1-b/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt
maxRatePerEndpoint: 100
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0742-4tvuu-psm-grpc-server
maxRatePerEndpoint: 100
...

I0623 00:45:35.642071 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506334321-5fec72f9d0af7-aa4fd886-3e04ad95
I0623 00:46:30.760337 139642840954624 compute.py:433] Backend psm-interop-server-20230623-0742-4tvuu-psm-grpc-server in zone us-central1-a reported healthy
I0623 00:46:44.352713 139642840954624 compute.py:433] Backend psm-interop-server-20230623-0742-4tvuu-psm-grpc-server-alt in zone us-west1-b reported healthy
I0623 00:46:48.357840 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.06_add_server_backends_to_backend_services ---
I0623 00:46:48.358130 139642840954624 xds_k8s_testcase.py:239] --- Starting subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0623 00:46:48.358345 139642840954624 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0742-4tvuu: server_target=xds:///psm-grpc-server:20929 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 00:46:48.358652 139642840954624 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0742-4tvuu%22;timeRange=2023-06-23T07:46:48.358462Z%2F2023-06-23T08:16:48.358462Z?project=grpc-testing
I0623 00:46:48.360948 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0742-4tvuu
labels:
name: psm-interop-client-20230623-0742-4tvuu
owner: xds-k8s-interop-test
...

I0623 00:46:48.420161 139642840954624 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0742-4tvuu created
I0623 00:46:48.420492 139642840954624 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0742-4tvuu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:46:48.896378 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0742-4tvuu
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:46:48.931004 139642840954624 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 00:46:48.940668 139642840954624 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0742-4tvuu
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y7eirl13
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y7eirl13
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-y7eirl13
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:20929"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:46:48.991819 139642840954624 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 00:46:48.992131 139642840954624 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 00:46:49.052571 139642840954624 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-66b7b78857-kr8cc']
I0623 00:46:49.052897 139642840954624 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-66b7b78857-kr8cc to start
I0623 00:47:03.315348 139642840954624 k8s_base_runner.py:554] Pod psm-grpc-client-66b7b78857-kr8cc ready, IP: 10.120.8.87
I0623 00:47:03.315696 139642840954624 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-66b7b78857-kr8cc to failover_test/test_app_logs/psm-interop-client-20230623-0742-4tvuu_psm-grpc-client-66b7b78857-kr8cc.log
I0623 00:47:03.316167 139642616829696 k8s_log_collector.py:70] Starting log collection thread 139642616829696 for psm-grpc-client-66b7b78857-kr8cc
I0623 00:47:03.316330 139642840954624 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 00:52:04.285252 139642840954624 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 07:46:49+00:00
last_update_time: 2023-06-23 07:46:49+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 07:46:49+00:00
last_update_time: 2023-06-23 07:46:49+00:00
message: ReplicaSet "psm-grpc-client-66b7b78857" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 00:52:04.285700 139642840954624 xds_k8s_testcase.py:244] --- Finished subTest __main__.FailoverTest.test_failover.07_start_test_client ---
I0623 00:52:04.285852 139642840954624 xds_k8s_testcase.py:640] ----- TestMethod __main__.FailoverTest.test_failover teardown -----
I0623 00:52:04.492197 139642840954624 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0742-4tvuu"
I0623 00:52:05.248587 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506724522-5fec746df08aa-83fbcf22-6febf645
I0623 00:52:24.931836 139642840954624 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0742-4tvuu"
I0623 00:52:25.055225 139642840954624 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:52:25.055482 139642840954624 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0742-4tvuu"
I0623 00:52:25.454566 139642840954624 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:52:25.454890 139642840954624 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0742-4tvuu"
I0623 00:52:25.831943 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506745476-5fec7481ec5f6-3fc7d802-4b5fab7d
I0623 00:52:28.140585 139642840954624 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0742-4tvuu"
I0623 00:52:28.449605 139642840954624 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:52:28.449882 139642840954624 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0742-4tvuu"
I0623 00:52:29.092204 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506748478-5fec7484c94e2-eaf1c136-adf004a6
I0623 00:52:31.462627 139642840954624 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0742-4tvuu"
I0623 00:52:31.851783 139642840954624 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:52:31.852049 139642840954624 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0742-4tvuu"
I0623 00:52:32.347392 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506751876-5fec748806ce6-1409bbcb-d2b69169
I0623 00:53:00.614360 139642840954624 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0742-4tvuu"
I0623 00:53:01.017831 139642840954624 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:53:01.018105 139642840954624 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0742-4tvuu"
I0623 00:53:01.527126 139642840954624 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0742-4tvuu" not deleted since it does not exist
I0623 00:53:01.527396 139642840954624 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0742-4tvuu"
I0623 00:53:02.135057 139642840954624 compute.py:535] Waiting 600 sec for compute operation id: operation-1687506781614-5fec74a4632ff-552db022-1d601657
I0623 00:53:06.650649 139642840954624 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:53:16.789985 139642840954624 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0742-4tvuu/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:53:17.303827 139642840954624 k8s_base_runner.py:487] Deleting service account psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:53:18.474206 139642840954624 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0742-4tvuu
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:53:48.642619 139642840954624 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:53:58.801550 139642840954624 k8s_base_runner.py:474] Deleting service psm-grpc-server
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:53:59.920163 139642840954624 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0742-4tvuu/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:54:00.410883 139642840954624 k8s_base_runner.py:487] Deleting service account psm-grpc-server
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:54:00.517665 139642840954624 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0742-4tvuu
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:56:30.929599 139642840954624 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:56:41.434561 139642840954624 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:56:41.593642 139642840954624 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0742-4tvuu/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:56:41.992289 139642840954624 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:56:42.157329 139642840954624 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0742-4tvuu
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 00:58:13.555155 139642840954624 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 00:58:13.555695 139642840954624 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-y7eirl13:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0742-4tvuu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-y7eirl13%22;timeRange=2023-06-23T07:46:48.358451Z%2F2023-06-23T07:53:48.634451Z?project=grpc-testing
I0623 00:58:13.555878 139642840954624 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-npf1aae2:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0742-4tvuu%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-npf1aae2%22;timeRange=2023-06-23T07:44:07.190471Z%2F2023-06-23T07:56:30.925518Z?project=grpc-testing
[ FAILED ] FailoverTest.test_failover
[ FAILED ] FailoverTest.test_failover
======================================================================
ERROR: test_failover (__main__.FailoverTest)
FailoverTest.test_failover
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/failover_test.py", line 114, in test_failover
test_client = self.startTestClient(default_test_servers[0])
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_failover (__main__.FailoverTest)
FailoverTest.test_failover
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 953.056s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test remove_neg_test
+ local test_name=remove_neg_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test'
+ python3 -m tests.remove_neg_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 00:58:14.755532 140453597599488 xds_k8s_testcase.py:130] ----- Testing RemoveNegTest -----
I0623 00:58:14.756010 140453597599488 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 00:58:14.756326 140453597599488 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 00:58:14.780350 140453597599488 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 00:58:14.803695 140453597599488 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RemoveNegTest.test_remove_neg
I0623 00:58:14.810061 140453597599488 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0758-1ns50
I0623 00:58:15.078902 140453597599488 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/remove_neg_test
I0623 00:58:15.364164 140453597599488 xds_k8s_testcase.py:625] Found unused xds port: 48934
I0623 00:58:15.366702 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0623 00:58:15.366942 140453597599488 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0758-1ns50"
I0623 00:58:15.370499 140453597599488 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0758-1ns50
type: GRPC
...

I0623 00:58:15.869819 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507095396-5fec75cfa2148-960e2d15-47648750
I0623 00:58:18.278188 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.00_create_health_check ---
I0623 00:58:18.278469 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0623 00:58:18.278570 140453597599488 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0758-1ns50"
I0623 00:58:18.286612 140453597599488 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0758-1ns50
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0758-1ns50
protocol: GRPC
...

I0623 00:58:19.087341 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507098313-5fec75d26a1d4-b28e7bda-6b85f535
I0623 00:58:53.686731 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.01_create_backend_services ---
I0623 00:58:53.686981 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0623 00:58:53.687084 140453597599488 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0758-1ns50": psm-grpc-server:48934 -> psm-interop-backend-service-20230623-0758-1ns50
I0623 00:58:53.698653 140453597599488 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0758-1ns50
hostRules:
- hosts:
- psm-grpc-server:48934
pathMatcher: psm-interop-path-matcher-20230623-0758-1ns50
name: psm-interop-url-map-20230623-0758-1ns50
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0758-1ns50
name: psm-interop-path-matcher-20230623-0758-1ns50
...

I0623 00:58:54.430298 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507133721-5fec75f42ecdd-c80e1280-034adf24
I0623 00:58:56.816574 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.02_create_url_map ---
I0623 00:58:56.816812 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0623 00:58:56.816913 140453597599488 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0758-1ns50 proxy "GRPC" to URL map psm-interop-url-map-20230623-0758-1ns50
I0623 00:58:56.819084 140453597599488 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0758-1ns50
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0758-1ns50
validate_for_proxyless: true
...

I0623 00:58:57.325320 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507136845-5fec75f7297dc-9b5c906a-6280ce13
I0623 00:58:59.652677 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.03_create_target_proxy ---
I0623 00:58:59.652930 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0623 00:58:59.653046 140453597599488 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0758-1ns50" in network "default-vpc": 0.0.0.0:48934 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0758-1ns50
I0623 00:58:59.655922 140453597599488 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0758-1ns50
network: global/networks/default-vpc
portRange: 48934
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0758-1ns50
...

I0623 00:59:00.471392 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507139683-5fec75f9de5ca-bd299a20-c29bca82
I0623 00:59:15.686902 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.04_create_forwarding_rule ---
I0623 00:59:15.687214 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0623 00:59:15.687331 140453597599488 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0758-1ns50: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:59:15.687629 140453597599488 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0758-1ns50%22;timeRange=2023-06-23T07:59:15.687437Z%2F2023-06-23T08:29:15.687437Z?project=grpc-testing
I0623 00:59:15.693811 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0758-1ns50
labels:
name: psm-interop-server-20230623-0758-1ns50
owner: xds-k8s-interop-test
...

I0623 00:59:15.754363 140453597599488 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0758-1ns50 created
I0623 00:59:15.758733 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0758-1ns50
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0758-1ns50-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:59:15.795597 140453597599488 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 00:59:15.795847 140453597599488 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 00:59:15.837023 140453597599488 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0758-1ns50-psm-grpc-server in zones=['us-central1-a']
I0623 00:59:15.837299 140453597599488 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0758-1ns50/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:59:16.384032 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0758-1ns50
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:59:16.420935 140453597599488 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 00:59:16.430561 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0758-1ns50
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lgb5v461
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lgb5v461
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-lgb5v461
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0758-1ns50-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:59:16.486644 140453597599488 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 00:59:16.486923 140453597599488 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 1 pod(s)
I0623 00:59:16.558712 140453597599488 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 1 pod(s): ['psm-grpc-server-768f5cd879-bbvcm']
I0623 00:59:16.559013 140453597599488 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-768f5cd879-bbvcm to start
I0623 00:59:19.674157 140453597599488 k8s_base_runner.py:554] Pod psm-grpc-server-768f5cd879-bbvcm ready, IP: 10.120.4.93
I0623 00:59:19.674536 140453597599488 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-768f5cd879-bbvcm to remove_neg_test/test_app_logs/psm-interop-server-20230623-0758-1ns50_psm-grpc-server-768f5cd879-bbvcm.log
I0623 00:59:19.675060 140453407569664 k8s_log_collector.py:70] Starting log collection thread 140453407569664 for psm-grpc-server-768f5cd879-bbvcm
I0623 00:59:19.675263 140453597599488 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 1 available replica(s)
I0623 00:59:27.819415 140453597599488 k8s_base_runner.py:522] Deployment psm-grpc-server has 1 replicas available
I0623 00:59:27.819855 140453597599488 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server-alt" to k8s namespace psm-interop-server-20230623-0758-1ns50: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=1
I0623 00:59:27.820088 140453597599488 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-alt:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server-alt%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0758-1ns50%22;timeRange=2023-06-23T07:59:27.819962Z%2F2023-06-23T08:29:27.819962Z?project=grpc-testing
I0623 00:59:27.856640 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0758-1ns50
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server-alt
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 00:59:27.873641 140453597599488 k8s_base_runner.py:280] Service psm-grpc-server-alt created
I0623 00:59:27.873915 140453597599488 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server-alt
I0623 00:59:27.920314 140453597599488 k8s_base_runner.py:601] Service psm-grpc-server-alt: detected NEG=psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt in zones=['us-central1-a']
I0623 00:59:27.920543 140453597599488 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0758-1ns50/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 00:59:28.282309 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0758-1ns50
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 00:59:28.292896 140453597599488 k8s_base_runner.py:280] ServiceAccount psm-grpc-server-alt created
I0623 00:59:28.301617 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server-alt
namespace: psm-interop-server-20230623-0758-1ns50
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-4mzpv71x
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-4mzpv71x
template:
metadata:
labels:
app: psm-grpc-server-alt
deployment_id: psm-grpc-server-alt-4mzpv71x
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server-alt
containers:
- name: psm-grpc-server-alt
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 00:59:28.325400 140453597599488 k8s_base_runner.py:280] Deployment psm-grpc-server-alt created
I0623 00:59:28.325675 140453597599488 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server-alt to initialize 1 pod(s)
I0623 00:59:29.364432 140453597599488 k8s_base_runner.py:541] Deployment psm-grpc-server-alt initialized 1 pod(s): ['psm-grpc-server-alt-7fdbfcc65f-gf94c']
I0623 00:59:29.364738 140453597599488 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-alt-7fdbfcc65f-gf94c to start
I0623 00:59:31.450069 140453597599488 k8s_base_runner.py:554] Pod psm-grpc-server-alt-7fdbfcc65f-gf94c ready, IP: 10.120.5.219
I0623 00:59:31.450399 140453597599488 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-alt-7fdbfcc65f-gf94c to remove_neg_test/test_app_logs/psm-interop-server-20230623-0758-1ns50_psm-grpc-server-alt-7fdbfcc65f-gf94c.log
I0623 00:59:31.451325 140453398914816 k8s_log_collector.py:70] Starting log collection thread 140453398914816 for psm-grpc-server-alt-7fdbfcc65f-gf94c
I0623 00:59:31.451808 140453597599488 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server-alt to report 1 available replica(s)
I0623 00:59:38.582555 140453597599488 k8s_base_runner.py:522] Deployment psm-grpc-server-alt has 1 replicas available
I0623 00:59:38.582977 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.05_start_test_servers ---
I0623 00:59:38.583108 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0623 00:59:38.589688 140453597599488 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 00:59:38.668954 140453597599488 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0758-1ns50-psm-grpc-server" in zone us-central1-a
I0623 00:59:38.669225 140453597599488 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0758-1ns50: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0758-1ns50-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server', zone='us-central1-a')}
I0623 00:59:38.673512 140453597599488 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 00:59:39.597156 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507178697-5fec761f1313e-9c2a4255-8da250ff
I0623 01:00:42.666292 140453597599488 compute.py:433] Backend psm-interop-server-20230623-0758-1ns50-psm-grpc-server in zone us-central1-a reported healthy
I0623 01:00:46.682145 140453597599488 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 01:00:46.769231 140453597599488 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt" in zone us-central1-a
I0623 01:00:46.769498 140453597599488 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0758-1ns50: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0758-1ns50-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server', zone='us-central1-a'), ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt', zone='us-central1-a')}
I0623 01:00:46.774449 140453597599488 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server
maxRatePerEndpoint: 5
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt
maxRatePerEndpoint: 5
...

I0623 01:00:47.688894 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507246797-5fec766005429-2afd46a0-bfd555d0
I0623 01:01:46.560663 140453597599488 compute.py:433] Backend psm-interop-server-20230623-0758-1ns50-psm-grpc-server in zone us-central1-a reported healthy
I0623 01:02:12.057517 140453597599488 compute.py:433] Backend psm-interop-server-20230623-0758-1ns50-psm-grpc-server-alt in zone us-central1-a reported healthy
I0623 01:02:16.062704 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.06_add_server_backends_to_backend_services ---
I0623 01:02:16.063077 140453597599488 xds_k8s_testcase.py:239] --- Starting subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0623 01:02:16.063238 140453597599488 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0758-1ns50: server_target=xds:///psm-grpc-server:48934 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 01:02:16.063619 140453597599488 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0758-1ns50%22;timeRange=2023-06-23T08:02:16.063342Z%2F2023-06-23T08:32:16.063342Z?project=grpc-testing
I0623 01:02:16.066460 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0758-1ns50
labels:
name: psm-interop-client-20230623-0758-1ns50
owner: xds-k8s-interop-test
...

I0623 01:02:16.130199 140453597599488 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0758-1ns50 created
I0623 01:02:16.130614 140453597599488 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0758-1ns50/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:02:16.590636 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0758-1ns50
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 01:02:16.629240 140453597599488 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 01:02:16.640002 140453597599488 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0758-1ns50
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qesk44jx
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qesk44jx
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-qesk44jx
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:48934"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 01:02:16.710681 140453597599488 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 01:02:16.710983 140453597599488 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 01:02:16.760115 140453597599488 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-74df68f5dc-p9zmt']
I0623 01:02:16.760478 140453597599488 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-74df68f5dc-p9zmt to start
I0623 01:02:19.862472 140453597599488 k8s_base_runner.py:554] Pod psm-grpc-client-74df68f5dc-p9zmt ready, IP: 10.120.8.92
I0623 01:02:19.862803 140453597599488 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-74df68f5dc-p9zmt to remove_neg_test/test_app_logs/psm-interop-client-20230623-0758-1ns50_psm-grpc-client-74df68f5dc-p9zmt.log
I0623 01:02:19.863260 140453390259968 k8s_log_collector.py:70] Starting log collection thread 140453390259968 for psm-grpc-client-74df68f5dc-p9zmt
I0623 01:02:19.863425 140453597599488 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 01:07:20.847046 140453597599488 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 08:02:16+00:00
last_update_time: 2023-06-23 08:02:16+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 08:02:16+00:00
last_update_time: 2023-06-23 08:02:16+00:00
message: ReplicaSet "psm-grpc-client-74df68f5dc" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 01:07:20.847425 140453597599488 xds_k8s_testcase.py:244] --- Finished subTest __main__.RemoveNegTest.test_remove_neg.07_start_test_client ---
I0623 01:07:20.847561 140453597599488 xds_k8s_testcase.py:640] ----- TestMethod __main__.RemoveNegTest.test_remove_neg teardown -----
I0623 01:07:20.920515 140453597599488 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0758-1ns50"
I0623 01:07:21.932369 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507640962-5fec77d7ecdef-86a11bbc-8a671aee
I0623 01:07:39.448699 140453597599488 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0758-1ns50"
I0623 01:07:39.597606 140453597599488 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:07:39.597875 140453597599488 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0758-1ns50"
I0623 01:07:40.014708 140453597599488 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:07:40.014989 140453597599488 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0758-1ns50"
I0623 01:07:41.102861 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507660042-5fec77ea1f2c2-36e420ab-a78e4c51
I0623 01:07:43.408331 140453597599488 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0758-1ns50"
I0623 01:07:43.860813 140453597599488 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:07:43.861085 140453597599488 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0758-1ns50"
I0623 01:07:44.639384 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507663891-5fec77edcaaaf-33b4ca71-9914cfe3
I0623 01:07:47.035043 140453597599488 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0758-1ns50"
I0623 01:07:47.413245 140453597599488 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:07:47.413509 140453597599488 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0758-1ns50"
I0623 01:07:48.067156 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507667443-5fec77f12de81-addaa49b-14ecc7c9
I0623 01:08:16.228326 140453597599488 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0758-1ns50"
I0623 01:08:16.641963 140453597599488 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:08:16.642371 140453597599488 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0758-1ns50"
I0623 01:08:17.014899 140453597599488 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0758-1ns50" not deleted since it does not exist
I0623 01:08:17.015148 140453597599488 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0758-1ns50"
I0623 01:08:17.695067 140453597599488 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507697139-5fec780d8010b-97cd657c-1cbd0c69
I0623 01:08:22.143063 140453597599488 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 01:08:32.244371 140453597599488 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0758-1ns50/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:08:32.751602 140453597599488 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 01:08:32.825163 140453597599488 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0758-1ns50
I0623 01:09:02.927745 140453597599488 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 01:09:13.026376 140453597599488 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 01:09:14.106209 140453597599488 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0758-1ns50/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:09:14.488444 140453597599488 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 01:09:14.559750 140453597599488 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0758-1ns50
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/exceptions.py:91: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
self.headers = http_resp.getheaders()
I0623 01:10:14.712123 140453597599488 k8s_base_runner.py:462] Deleting deployment psm-grpc-server-alt
I0623 01:10:14.745100 140453597599488 k8s_base_runner.py:466] Deployment psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: deployments.apps "psm-grpc-server-alt" not found
I0623 01:10:14.745560 140453597599488 k8s_base_runner.py:474] Deleting service psm-grpc-server-alt
I0623 01:10:14.780220 140453597599488 k8s_base_runner.py:478] Service psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: services "psm-grpc-server-alt" not found
I0623 01:10:14.780567 140453597599488 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0758-1ns50/psm-grpc-server-alt] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:10:15.198812 140453597599488 k8s_base_runner.py:487] Deleting service account psm-grpc-server-alt
I0623 01:10:15.230724 140453597599488 k8s_base_runner.py:491] Service account psm-grpc-server-alt deletion failed: Kubernetes API returned 404 Not Found: serviceaccounts "psm-grpc-server-alt" not found
I0623 01:10:15.231231 140453597599488 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0758-1ns50
I0623 01:10:15.236643 140453597599488 k8s_base_runner.py:503] Namespace psm-interop-server-20230623-0758-1ns50 deletion failed: Kubernetes API returned 404 Not Found: namespaces "psm-interop-server-20230623-0758-1ns50" not found
I0623 01:10:15.237252 140453597599488 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 01:10:15.237602 140453597599488 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-qesk44jx:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0758-1ns50%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-qesk44jx%22;timeRange=2023-06-23T08:02:16.063327Z%2F2023-06-23T08:09:02.924999Z?project=grpc-testing
I0623 01:10:15.237784 140453597599488 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-lgb5v461:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0758-1ns50%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-lgb5v461%22;timeRange=2023-06-23T07:59:15.687425Z%2F2023-06-23T08:10:14.711779Z?project=grpc-testing
[ FAILED ] RemoveNegTest.test_remove_neg
[ FAILED ] RemoveNegTest.test_remove_neg
======================================================================
ERROR: test_remove_neg (__main__.RemoveNegTest)
RemoveNegTest.test_remove_neg
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/remove_neg_test.py", line 89, in test_remove_neg
test_client = self.startTestClient(default_test_servers[0])
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_remove_neg (__main__.RemoveNegTest)
RemoveNegTest.test_remove_neg
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 720.488s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test round_robin_test
+ local test_name=round_robin_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test'
+ python3 -m tests.round_robin_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 01:10:16.548408 140248609728256 xds_k8s_testcase.py:130] ----- Testing RoundRobinTest -----
I0623 01:10:16.548898 140248609728256 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 01:10:16.549235 140248609728256 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 01:10:16.575235 140248609728256 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 01:10:16.599466 140248609728256 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] RoundRobinTest.test_round_robin
I0623 01:10:16.606831 140248609728256 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0810-8m8iw
I0623 01:10:16.919638 140248609728256 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/round_robin_test
I0623 01:10:17.122651 140248609728256 xds_k8s_testcase.py:625] Found unused xds port: 28269
I0623 01:10:17.123303 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0623 01:10:17.123461 140248609728256 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0810-8m8iw"
I0623 01:10:17.127439 140248609728256 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0810-8m8iw
type: GRPC
...

I0623 01:10:17.694482 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507817151-5fec787ff3b3a-c55bbc85-bc06c658
I0623 01:10:20.034513 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.00_create_health_check ---
I0623 01:10:20.034852 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0623 01:10:20.034979 140248609728256 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0810-8m8iw"
I0623 01:10:20.044358 140248609728256 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0810-8m8iw
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0810-8m8iw
protocol: GRPC
...

I0623 01:10:20.840563 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507820067-5fec7882bb989-7906ef48-3c195718
I0623 01:11:04.195098 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.01_create_backend_services ---
I0623 01:11:04.195363 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0623 01:11:04.195477 140248609728256 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0810-8m8iw": psm-grpc-server:28269 -> psm-interop-backend-service-20230623-0810-8m8iw
I0623 01:11:04.209114 140248609728256 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0810-8m8iw
hostRules:
- hosts:
- psm-grpc-server:28269
pathMatcher: psm-interop-path-matcher-20230623-0810-8m8iw
name: psm-interop-url-map-20230623-0810-8m8iw
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0810-8m8iw
name: psm-interop-path-matcher-20230623-0810-8m8iw
...

I0623 01:11:04.927038 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507864234-5fec78acdaa72-c93faafa-2458f282
I0623 01:11:07.300927 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.02_create_url_map ---
I0623 01:11:07.301183 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0623 01:11:07.301313 140248609728256 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0810-8m8iw proxy "GRPC" to URL map psm-interop-url-map-20230623-0810-8m8iw
I0623 01:11:07.303563 140248609728256 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0810-8m8iw
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0810-8m8iw
validate_for_proxyless: true
...

I0623 01:11:08.020028 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507867329-5fec78afce264-cf7b1c95-6a2ee1da
I0623 01:11:10.337508 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.03_create_target_proxy ---
I0623 01:11:10.337788 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0623 01:11:10.337921 140248609728256 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0810-8m8iw" in network "default-vpc": 0.0.0.0:28269 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0810-8m8iw
I0623 01:11:10.341006 140248609728256 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0810-8m8iw
network: global/networks/default-vpc
portRange: 28269
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0810-8m8iw
...

I0623 01:11:11.202374 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507870366-5fec78b2b3a30-c7695ca6-32bbc5d9
I0623 01:11:26.453985 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.04_create_forwarding_rule ---
I0623 01:11:26.454316 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0623 01:11:26.454473 140248609728256 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0810-8m8iw: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=2
I0623 01:11:26.454793 140248609728256 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0810-8m8iw%22;timeRange=2023-06-23T08:11:26.454598Z%2F2023-06-23T08:41:26.454598Z?project=grpc-testing
I0623 01:11:26.462035 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0810-8m8iw
labels:
name: psm-interop-server-20230623-0810-8m8iw
owner: xds-k8s-interop-test
...

I0623 01:11:26.524123 140248609728256 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0810-8m8iw created
I0623 01:11:26.527895 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0810-8m8iw
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0810-8m8iw-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 01:11:26.576901 140248609728256 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 01:11:26.577170 140248609728256 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 01:11:26.620529 140248609728256 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0810-8m8iw-psm-grpc-server in zones=['us-central1-a']
I0623 01:11:26.620791 140248609728256 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0810-8m8iw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:11:27.121874 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0810-8m8iw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 01:11:27.160736 140248609728256 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 01:11:27.169791 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0810-8m8iw
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ngw4jnq5
owner: xds-k8s-interop-test
spec:
replicas: 2
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ngw4jnq5
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-ngw4jnq5
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:v1.46.x
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0810-8m8iw-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 01:11:27.227976 140248609728256 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 01:11:27.228268 140248609728256 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 2 pod(s)
I0623 01:11:28.312236 140248609728256 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 2 pod(s): ['psm-grpc-server-cffc66c8c-4rlx8', 'psm-grpc-server-cffc66c8c-7rqnp']
I0623 01:11:28.312587 140248609728256 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-cffc66c8c-4rlx8 to start
I0623 01:11:30.391564 140248609728256 k8s_base_runner.py:554] Pod psm-grpc-server-cffc66c8c-4rlx8 ready, IP: 10.120.5.224
I0623 01:11:30.391903 140248609728256 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-cffc66c8c-4rlx8 to round_robin_test/test_app_logs/psm-interop-server-20230623-0810-8m8iw_psm-grpc-server-cffc66c8c-4rlx8.log
I0623 01:11:30.392434 140248419698432 k8s_log_collector.py:70] Starting log collection thread 140248419698432 for psm-grpc-server-cffc66c8c-4rlx8
I0623 01:11:30.392600 140248609728256 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-cffc66c8c-7rqnp to start
I0623 01:11:32.478947 140248609728256 k8s_base_runner.py:554] Pod psm-grpc-server-cffc66c8c-7rqnp ready, IP: 10.120.3.31
I0623 01:11:32.479485 140248609728256 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-cffc66c8c-7rqnp to round_robin_test/test_app_logs/psm-interop-server-20230623-0810-8m8iw_psm-grpc-server-cffc66c8c-7rqnp.log
I0623 01:11:32.480024 140248411305728 k8s_log_collector.py:70] Starting log collection thread 140248411305728 for psm-grpc-server-cffc66c8c-7rqnp
I0623 01:11:32.480322 140248609728256 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 2 available replica(s)
I0623 01:11:37.591009 140248609728256 k8s_base_runner.py:522] Deployment psm-grpc-server has 2 replicas available
I0623 01:11:37.591596 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.05_start_test_servers ---
I0623 01:11:37.591753 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0623 01:11:37.599238 140248609728256 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 01:11:37.673439 140248609728256 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0810-8m8iw-psm-grpc-server" in zone us-central1-a
I0623 01:11:37.673683 140248609728256 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0810-8m8iw: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0810-8m8iw-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0810-8m8iw-psm-grpc-server', zone='us-central1-a')}
I0623 01:11:37.678053 140248609728256 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0810-8m8iw-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 01:11:38.533661 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687507897702-5fec78ccc5911-e74626da-465c058d
I0623 01:12:34.906305 140248609728256 compute.py:433] Backend psm-interop-server-20230623-0810-8m8iw-psm-grpc-server in zone us-central1-a reported healthy
I0623 01:12:38.911719 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.06_add_server_backends_to_backend_services ---
I0623 01:12:38.912045 140248609728256 xds_k8s_testcase.py:239] --- Starting subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0623 01:12:38.912174 140248609728256 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0810-8m8iw: server_target=xds:///psm-grpc-server:28269 rpc=UnaryCall qps=25 metadata='' secure_mode=False print_response=False
I0623 01:12:38.912406 140248609728256 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0810-8m8iw%22;timeRange=2023-06-23T08:12:38.912262Z%2F2023-06-23T08:42:38.912262Z?project=grpc-testing
I0623 01:12:38.914738 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0810-8m8iw
labels:
name: psm-interop-client-20230623-0810-8m8iw
owner: xds-k8s-interop-test
...

I0623 01:12:38.928701 140248609728256 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0810-8m8iw created
I0623 01:12:38.928985 140248609728256 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0810-8m8iw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:12:40.725162 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0810-8m8iw
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 01:12:40.761519 140248609728256 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 01:12:40.771841 140248609728256 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0810-8m8iw
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-l6sexdnq
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-l6sexdnq
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-l6sexdnq
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:28269"
- "--stats_port=8079"
- "--qps=25"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 01:12:40.821047 140248609728256 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 01:12:40.821351 140248609728256 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 01:12:41.872680 140248609728256 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-99d464985-js6qt']
I0623 01:12:41.873006 140248609728256 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-99d464985-js6qt to start
I0623 01:12:43.946269 140248609728256 k8s_base_runner.py:554] Pod psm-grpc-client-99d464985-js6qt ready, IP: 10.120.8.96
I0623 01:12:43.946568 140248609728256 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-99d464985-js6qt to round_robin_test/test_app_logs/psm-interop-client-20230623-0810-8m8iw_psm-grpc-client-99d464985-js6qt.log
I0623 01:12:43.947028 140248331843328 k8s_log_collector.py:70] Starting log collection thread 140248331843328 for psm-grpc-client-99d464985-js6qt
I0623 01:12:43.947211 140248609728256 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 01:17:44.923079 140248609728256 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 08:12:40+00:00
last_update_time: 2023-06-23 08:12:40+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 08:12:40+00:00
last_update_time: 2023-06-23 08:12:40+00:00
message: ReplicaSet "psm-grpc-client-99d464985" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 01:17:44.923535 140248609728256 xds_k8s_testcase.py:244] --- Finished subTest __main__.RoundRobinTest.test_round_robin.07_start_test_client ---
I0623 01:17:44.923684 140248609728256 xds_k8s_testcase.py:640] ----- TestMethod __main__.RoundRobinTest.test_round_robin teardown -----
I0623 01:17:45.004775 140248609728256 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0810-8m8iw"
I0623 01:17:45.733452 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508265032-5fec7a2b1597c-58db3935-73186d7b
I0623 01:18:05.269070 140248609728256 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0810-8m8iw"
I0623 01:18:05.441967 140248609728256 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:05.442331 140248609728256 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0810-8m8iw"
I0623 01:18:05.862984 140248609728256 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:05.863281 140248609728256 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0810-8m8iw"
I0623 01:18:06.455694 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508285877-5fec7a3ef6ca9-a34e0158-c6bd8646
I0623 01:18:08.751137 140248609728256 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0810-8m8iw"
I0623 01:18:09.103863 140248609728256 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:09.104116 140248609728256 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0810-8m8iw"
I0623 01:18:09.822373 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508289125-5fec7a420fd0b-a4ea5153-7021de00
I0623 01:18:12.183068 140248609728256 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0810-8m8iw"
I0623 01:18:12.514043 140248609728256 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:12.514387 140248609728256 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0810-8m8iw"
I0623 01:18:13.201794 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508292535-5fec7a45503c4-5c92b92a-5160ddb5
I0623 01:18:43.617718 140248609728256 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0810-8m8iw"
I0623 01:18:44.038167 140248609728256 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:44.038481 140248609728256 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0810-8m8iw"
I0623 01:18:44.421747 140248609728256 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0810-8m8iw" not deleted since it does not exist
I0623 01:18:44.422006 140248609728256 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0810-8m8iw"
I0623 01:18:44.968928 140248609728256 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508324516-5fec7a63d02fb-7baf6986-bd4347bd
I0623 01:18:49.495556 140248609728256 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 01:18:59.560006 140248609728256 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0810-8m8iw/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:19:00.035396 140248609728256 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 01:19:00.103126 140248609728256 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0810-8m8iw
I0623 01:19:30.257883 140248609728256 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 01:19:40.355246 140248609728256 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 01:19:41.439489 140248609728256 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0810-8m8iw/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:19:41.793350 140248609728256 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 01:19:41.865164 140248609728256 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0810-8m8iw
I0623 01:22:12.185544 140248609728256 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 01:22:12.186092 140248609728256 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-l6sexdnq:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0810-8m8iw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-l6sexdnq%22;timeRange=2023-06-23T08:12:38.912254Z%2F2023-06-23T08:19:30.253145Z?project=grpc-testing
I0623 01:22:12.186365 140248609728256 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-ngw4jnq5:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0810-8m8iw%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-ngw4jnq5%22;timeRange=2023-06-23T08:11:26.454584Z%2F2023-06-23T08:22:12.185338Z?project=grpc-testing
[ FAILED ] RoundRobinTest.test_round_robin
[ FAILED ] RoundRobinTest.test_round_robin
======================================================================
ERROR: test_round_robin (__main__.RoundRobinTest)
RoundRobinTest.test_round_robin
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/round_robin_test.py", line 58, in test_round_robin
test_client = self.startTestClient(test_servers[0])
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_round_robin (__main__.RoundRobinTest)
RoundRobinTest.test_round_robin
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 715.643s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ for test in '"${test_suites[@]}"'
+ run_test outlier_detection_test
+ local test_name=outlier_detection_test
+ local out_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test
+ mkdir -pv /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test
mkdir: created directory '/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test'
+ python3 -m tests.outlier_detection_test --flagfile=config/grpc-testing.cfg --kube_context=gke_grpc-testing_us-central1-a_psm-interop-lb-primary --secondary_kube_context=gke_grpc-testing_us-west1-b_psm-interop-lb-secondary --client_image=gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229 --server_image=gcr.io/grpc-testing/xds-interop/java-server:v1.46.x --testing_version=dev-v1.8.x --force_cleanup --collect_app_logs --log_dir=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test --xml_output_file=/tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test/sponge_log.xml
+ tee /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test/sponge_log.log
Running tests under Python 3.9.5: /home/kbuilder/.pyenv/versions/k8s_xds_test_runner/bin/python3
I0623 01:22:13.432210 140564729067264 xds_k8s_testcase.py:130] ----- Testing OutlierDetectionTest -----
I0623 01:22:13.433012 140564729067264 xds_k8s_testcase.py:131] Logs timezone: PDT
I0623 01:22:13.433848 140564729067264 skips.py:124] Detected language and version: TestConfig(client_lang='node', server_lang='java', version='dev-v1.8.x')
I0623 01:22:13.458989 140564729067264 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-central1-a_psm-interop-lb-primary", active host: https://35.238.98.54
I0623 01:22:13.485493 140564729067264 k8s.py:129] Using kubernetes context "gke_grpc-testing_us-west1-b_psm-interop-lb-secondary", active host: https://35.227.140.120
[ RUN ] OutlierDetectionTest.test_outlier_detection
I0623 01:22:13.492378 140564729067264 xds_k8s_testcase.py:590] Test run resource prefix: psm-interop, suffix: 20230623-0822-7gihn
I0623 01:22:13.831822 140564729067264 logs.py:31] Log root dir: /tmpfs/src/artifacts/grpc/node/v1.8.x/xds_k8s_lb/outlier_detection_test
I0623 01:22:14.267357 140564729067264 xds_k8s_testcase.py:625] Found unused xds port: 63842
I0623 01:22:14.267818 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0623 01:22:14.267969 140564729067264 traffic_director.py:187] Creating GRPC Health Check "psm-interop-health-check-20230623-0822-7gihn"
I0623 01:22:14.271664 140564729067264 compute.py:476] Creating compute resource:
---
grpcHealthCheck:
portSpecification: USE_SERVING_PORT
name: psm-interop-health-check-20230623-0822-7gihn
type: GRPC
...

I0623 01:22:14.873929 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508534283-5fec7b2bdceaa-b0bc1644-a635f39a
I0623 01:22:17.171131 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.00_create_health_check ---
I0623 01:22:17.171387 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0623 01:22:17.171495 140564729067264 traffic_director.py:214] Creating GRPC Backend Service "psm-interop-backend-service-20230623-0822-7gihn"
I0623 01:22:17.180152 140564729067264 compute.py:476] Creating compute resource:
---
healthChecks:
- https://www.googleapis.com/compute/v1/projects/grpc-testing/global/healthChecks/psm-interop-health-check-20230623-0822-7gihn
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-backend-service-20230623-0822-7gihn
outlierDetection:
interval:
nanos: 0
seconds: 2
successRateRequestVolume: 20
protocol: GRPC
...

I0623 01:22:17.834101 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508537196-5fec7b2ea3f16-45369fb5-6dabd254
I0623 01:23:16.557358 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.01_create_backend_service ---
I0623 01:23:16.557645 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0623 01:23:16.557753 140564729067264 traffic_director.py:470] Creating URL map "psm-interop-url-map-20230623-0822-7gihn": psm-grpc-server:63842 -> psm-interop-backend-service-20230623-0822-7gihn
I0623 01:23:16.571171 140564729067264 compute.py:476] Creating compute resource:
---
defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0822-7gihn
hostRules:
- hosts:
- psm-grpc-server:63842
pathMatcher: psm-interop-path-matcher-20230623-0822-7gihn
name: psm-interop-url-map-20230623-0822-7gihn
pathMatchers:
- defaultService: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/backendServices/psm-interop-backend-service-20230623-0822-7gihn
name: psm-interop-path-matcher-20230623-0822-7gihn
...

I0623 01:23:17.240431 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508596583-5fec7b6746a93-91f4453d-501b05a4
I0623 01:23:19.523574 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.02_create_url_map ---
I0623 01:23:19.523875 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0623 01:23:19.523986 140564729067264 traffic_director.py:569] Creating target psm-interop-target-proxy-20230623-0822-7gihn proxy "GRPC" to URL map psm-interop-url-map-20230623-0822-7gihn
I0623 01:23:19.526152 140564729067264 compute.py:476] Creating compute resource:
---
name: psm-interop-target-proxy-20230623-0822-7gihn
url_map: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/urlMaps/psm-interop-url-map-20230623-0822-7gihn
validate_for_proxyless: true
...

I0623 01:23:20.305541 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508599538-5fec7b6a182ab-fb2ffc44-36d49ea4
I0623 01:23:22.611929 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.03_create_target_proxy ---
I0623 01:23:22.612209 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0623 01:23:22.612332 140564729067264 traffic_director.py:645] Creating forwarding rule "psm-interop-forwarding-rule-20230623-0822-7gihn" in network "default-vpc": 0.0.0.0:63842 -> https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0822-7gihn
I0623 01:23:22.615376 140564729067264 compute.py:476] Creating compute resource:
---
IPAddress: 0.0.0.0
loadBalancingScheme: INTERNAL_SELF_MANAGED
name: psm-interop-forwarding-rule-20230623-0822-7gihn
network: global/networks/default-vpc
portRange: 63842
target: https://www.googleapis.com/compute/v1/projects/grpc-testing/global/targetGrpcProxies/psm-interop-target-proxy-20230623-0822-7gihn
...

I0623 01:23:23.391001 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508602626-5fec7b6d0a3a5-764082b9-13f21374
I0623 01:24:28.462545 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.04_create_forwarding_rule ---
I0623 01:24:28.462825 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0623 01:24:28.462956 140564729067264 k8s_xds_server_runner.py:144] Deploying xDS test server "psm-grpc-server" to k8s namespace psm-interop-server-20230623-0822-7gihn: test_port=8080 maintenance_port=8080 secure_mode=False replica_count=5
I0623 01:24:28.463288 140564729067264 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0822-7gihn%22;timeRange=2023-06-23T08:24:28.463060Z%2F2023-06-23T08:54:28.463060Z?project=grpc-testing
I0623 01:24:28.469402 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-server-20230623-0822-7gihn
labels:
name: psm-interop-server-20230623-0822-7gihn
owner: xds-k8s-interop-test
...

I0623 01:24:28.538448 140564729067264 k8s_base_runner.py:280] Namespace psm-interop-server-20230623-0822-7gihn created
I0623 01:24:28.542258 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.service.yaml:
---
apiVersion: v1
kind: Service
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0822-7gihn
labels:
owner: xds-k8s-interop-test
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8080":{"name":"psm-interop-server-20230623-0822-7gihn-psm-grpc-server"}}}'
spec:
type: ClusterIP
selector:
app: psm-grpc-server
ports:
- port: 8080
protocol: TCP
targetPort: 8080
...

I0623 01:24:28.587715 140564729067264 k8s_base_runner.py:280] Service psm-grpc-server created
I0623 01:24:28.587975 140564729067264 k8s_base_runner.py:596] Waiting for NEG for service psm-grpc-server
I0623 01:24:28.636261 140564729067264 k8s_base_runner.py:601] Service psm-grpc-server: detected NEG=psm-interop-server-20230623-0822-7gihn-psm-grpc-server in zones=['us-central1-a']
I0623 01:24:28.636502 140564729067264 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0822-7gihn/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:24:29.209223 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0822-7gihn
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 01:24:29.245794 140564729067264 k8s_base_runner.py:280] ServiceAccount psm-grpc-server created
I0623 01:24:29.254674 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/server.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-server
namespace: psm-interop-server-20230623-0822-7gihn
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nhr48cwk
owner: xds-k8s-interop-test
spec:
replicas: 5
selector:
matchLabels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nhr48cwk
template:
metadata:
labels:
app: psm-grpc-server
deployment_id: psm-grpc-server-nhr48cwk
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-server
containers:
- name: psm-grpc-server
image: gcr.io/grpc-testing/xds-interop/java-server:d56f8fbe1d2822bc4f91515dd471ad49493fc385
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8080
periodSeconds: 3
failureThreshold: 1000
args:
- "--port=8080"
ports:
- containerPort: 8080
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_V3_SUPPORT
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
- "--node-metadata=app=psm-interop-server-20230623-0822-7gihn-psm-grpc-server"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 01:24:29.309870 140564729067264 k8s_base_runner.py:280] Deployment psm-grpc-server created
I0623 01:24:29.310135 140564729067264 k8s_base_runner.py:531] Waiting for deployment psm-grpc-server to initialize 5 pod(s)
I0623 01:24:30.422849 140564729067264 k8s_base_runner.py:541] Deployment psm-grpc-server initialized 5 pod(s): ['psm-grpc-server-844f4674f6-hbzst', 'psm-grpc-server-844f4674f6-m85c6', 'psm-grpc-server-844f4674f6-ntqb4', 'psm-grpc-server-844f4674f6-sv6wd', 'psm-grpc-server-844f4674f6-vzfll']
I0623 01:24:30.423389 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-844f4674f6-hbzst to start
I0623 01:24:32.509708 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-server-844f4674f6-hbzst ready, IP: 10.120.7.249
I0623 01:24:32.510022 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-844f4674f6-hbzst to outlier_detection_test/test_app_logs/psm-interop-server-20230623-0822-7gihn_psm-grpc-server-844f4674f6-hbzst.log
I0623 01:24:32.510602 140564538775296 k8s_log_collector.py:70] Starting log collection thread 140564538775296 for psm-grpc-server-844f4674f6-hbzst
I0623 01:24:32.510772 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-844f4674f6-m85c6 to start
I0623 01:24:32.586678 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-server-844f4674f6-m85c6 ready, IP: 10.120.0.20
I0623 01:24:32.586994 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-844f4674f6-m85c6 to outlier_detection_test/test_app_logs/psm-interop-server-20230623-0822-7gihn_psm-grpc-server-844f4674f6-m85c6.log
I0623 01:24:32.587456 140564530382592 k8s_log_collector.py:70] Starting log collection thread 140564530382592 for psm-grpc-server-844f4674f6-m85c6
I0623 01:24:32.587618 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-844f4674f6-ntqb4 to start
I0623 01:24:33.682997 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-server-844f4674f6-ntqb4 ready, IP: 10.120.5.229
I0623 01:24:33.683434 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-844f4674f6-ntqb4 to outlier_detection_test/test_app_logs/psm-interop-server-20230623-0822-7gihn_psm-grpc-server-844f4674f6-ntqb4.log
I0623 01:24:33.683901 140564521989888 k8s_log_collector.py:70] Starting log collection thread 140564521989888 for psm-grpc-server-844f4674f6-ntqb4
I0623 01:24:33.684182 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-844f4674f6-sv6wd to start
I0623 01:24:33.731289 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-server-844f4674f6-sv6wd ready, IP: 10.120.4.100
I0623 01:24:33.732204 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-844f4674f6-sv6wd to outlier_detection_test/test_app_logs/psm-interop-server-20230623-0822-7gihn_psm-grpc-server-844f4674f6-sv6wd.log
I0623 01:24:33.732710 140564513597184 k8s_log_collector.py:70] Starting log collection thread 140564513597184 for psm-grpc-server-844f4674f6-sv6wd
I0623 01:24:33.732972 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-server-844f4674f6-vzfll to start
I0623 01:24:33.779930 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-server-844f4674f6-vzfll ready, IP: 10.120.2.204
I0623 01:24:33.780843 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-server-844f4674f6-vzfll to outlier_detection_test/test_app_logs/psm-interop-server-20230623-0822-7gihn_psm-grpc-server-844f4674f6-vzfll.log
I0623 01:24:33.781415 140564505204480 k8s_log_collector.py:70] Starting log collection thread 140564505204480 for psm-grpc-server-844f4674f6-vzfll
I0623 01:24:33.781579 140564729067264 k8s_base_runner.py:513] Waiting for deployment psm-grpc-server to report 5 available replica(s)
I0623 01:24:40.928962 140564729067264 k8s_base_runner.py:522] Deployment psm-grpc-server has 5 replicas available
I0623 01:24:40.929696 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.05_start_test_servers ---
I0623 01:24:40.929872 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0623 01:24:40.935519 140564729067264 traffic_director.py:246] Waiting for Network Endpoint Groups to load endpoints.
I0623 01:24:41.007285 140564729067264 traffic_director.py:249] Loaded NEG "psm-interop-server-20230623-0822-7gihn-psm-grpc-server" in zone us-central1-a
I0623 01:24:41.007552 140564729067264 traffic_director.py:268] Adding backends to Backend Service psm-interop-backend-service-20230623-0822-7gihn: {ComputeV1.ZonalGcpResource(name='psm-interop-server-20230623-0822-7gihn-psm-grpc-server', url='https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0822-7gihn-psm-grpc-server', zone='us-central1-a')}
I0623 01:24:41.011910 140564729067264 compute.py:483] Patching compute resource:
---
backends:
- balancingMode: RATE
group: https://www.googleapis.com/compute/v1/projects/grpc-testing/zones/us-central1-a/networkEndpointGroups/psm-interop-server-20230623-0822-7gihn-psm-grpc-server
maxRatePerEndpoint: 5
...

I0623 01:24:41.870962 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687508681027-5fec7bb7cf0b4-e670141c-49eb9f7b
I0623 01:25:49.213115 140564729067264 compute.py:433] Backend psm-interop-server-20230623-0822-7gihn-psm-grpc-server in zone us-central1-a reported healthy
I0623 01:25:53.218000 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.06_add_server_backends_to_backend_services ---
I0623 01:25:53.218385 140564729067264 xds_k8s_testcase.py:239] --- Starting subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0623 01:25:53.218538 140564729067264 k8s_xds_client_runner.py:107] Deploying xDS test client "psm-grpc-client" to k8s namespace psm-interop-client-20230623-0822-7gihn: server_target=xds:///psm-grpc-server:63842 rpc=UnaryCall qps=100 metadata='' secure_mode=False print_response=False
I0623 01:25:53.218782 140564729067264 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0822-7gihn%22;timeRange=2023-06-23T08:25:53.218629Z%2F2023-06-23T08:55:53.218629Z?project=grpc-testing
I0623 01:25:53.221711 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/namespace.yaml:
---
apiVersion: v1
kind: Namespace
metadata:
name: psm-interop-client-20230623-0822-7gihn
labels:
name: psm-interop-client-20230623-0822-7gihn
owner: xds-k8s-interop-test
...

I0623 01:25:53.237398 140564729067264 k8s_base_runner.py:280] Namespace psm-interop-client-20230623-0822-7gihn created
I0623 01:25:53.237679 140564729067264 k8s_base_runner.py:337] Granting roles/iam.workloadIdentityUser to serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0822-7gihn/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:25:53.700740 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/service-account.yaml:
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0822-7gihn
labels:
owner: xds-k8s-interop-test
annotations:
iam.gke.io/gcp-service-account: xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
...

I0623 01:25:53.738770 140564729067264 k8s_base_runner.py:280] ServiceAccount psm-grpc-client created
I0623 01:25:53.750614 140564729067264 k8s_base_runner.py:258] Rendered template kubernetes-manifests/client.deployment.yaml:
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: psm-grpc-client
namespace: psm-interop-client-20230623-0822-7gihn
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ou0x07j4
owner: xds-k8s-interop-test
spec:
replicas: 1
selector:
matchLabels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ou0x07j4
template:
metadata:
labels:
app: psm-grpc-client
deployment_id: psm-grpc-client-ou0x07j4
owner: xds-k8s-interop-test
spec:
serviceAccountName: psm-grpc-client
containers:
- name: psm-grpc-client
image: gcr.io/grpc-testing/xds-interop/node-client:9441de78f655ada34ada0dc1a8057122eb21f229
imagePullPolicy: Always
startupProbe:
tcpSocket:
port: 8079
periodSeconds: 3
failureThreshold: 1000
args:
- "--server=xds:///psm-grpc-server:63842"
- "--stats_port=8079"
- "--qps=100"
- "--rpc=UnaryCall"
- "--metadata="
- "--print_response=False"
ports:
- containerPort: 8079
env:
- name: GRPC_XDS_BOOTSTRAP
value: "/tmp/grpc-xds/td-grpc-bootstrap.json"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RING_HASH
value: "true"
- name: GRPC_XDS_EXPERIMENTAL_ENABLE_RETRY
value: "true"
- name: GRPC_EXPERIMENTAL_XDS_CUSTOM_LB_CONFIG
value: "true"
- name: GRPC_EXPERIMENTAL_ENABLE_OUTLIER_DETECTION
value: "true"
volumeMounts:
- mountPath: /tmp/grpc-xds/
name: grpc-td-conf
readOnly: true
resources:
limits:
cpu: 800m
memory: 512Mi
requests:
cpu: 100m
memory: 512Mi
initContainers:
- name: grpc-td-init
image: gcr.io/grpc-testing/td-grpc-bootstrap:d6baaf7b0e0c63054ac4d9bedc09021ff261d599
imagePullPolicy: Always
args:
- "--output=/tmp/bootstrap/td-grpc-bootstrap.json"
- "--vpc-network-name=default-vpc"
resources:
limits:
cpu: 100m
memory: 100Mi
requests:
cpu: 10m
memory: 100Mi
volumeMounts:
- mountPath: /tmp/bootstrap/
name: grpc-td-conf
volumes:
- name: grpc-td-conf
emptyDir:
medium: Memory
...

I0623 01:25:53.808606 140564729067264 k8s_base_runner.py:280] Deployment psm-grpc-client created
I0623 01:25:53.808920 140564729067264 k8s_base_runner.py:531] Waiting for deployment psm-grpc-client to initialize 1 pod(s)
I0623 01:25:53.876332 140564729067264 k8s_base_runner.py:541] Deployment psm-grpc-client initialized 1 pod(s): ['psm-grpc-client-5d75c5bcc7-wltrg']
I0623 01:25:53.876626 140564729067264 k8s_base_runner.py:551] Waiting for pod psm-grpc-client-5d75c5bcc7-wltrg to start
I0623 01:25:56.970298 140564729067264 k8s_base_runner.py:554] Pod psm-grpc-client-5d75c5bcc7-wltrg ready, IP: 10.120.5.230
I0623 01:25:56.970625 140564729067264 k8s_base_runner.py:577] Enabling log collection from pod psm-grpc-client-5d75c5bcc7-wltrg to outlier_detection_test/test_app_logs/psm-interop-client-20230623-0822-7gihn_psm-grpc-client-5d75c5bcc7-wltrg.log
I0623 01:25:56.971265 140564496549632 k8s_log_collector.py:70] Starting log collection thread 140564496549632 for psm-grpc-client-5d75c5bcc7-wltrg
I0623 01:25:56.971587 140564729067264 k8s_base_runner.py:513] Waiting for deployment psm-grpc-client to report 1 available replica(s)
E0623 01:30:57.031595 140564729067264 k8s.py:478] Timeout 0:05:00 (h:mm:ss) waiting for deployment psm-grpc-client to report 1 replicas available. Last status:
psm-grpc-client:
---
available_replicas: null
collision_count: null
conditions:
- last_transition_time: 2023-06-23 08:25:53+00:00
last_update_time: 2023-06-23 08:25:53+00:00
message: Deployment does not have minimum availability.
reason: MinimumReplicasUnavailable
status: 'False'
type: Available
- last_transition_time: 2023-06-23 08:25:53+00:00
last_update_time: 2023-06-23 08:25:53+00:00
message: ReplicaSet "psm-grpc-client-5d75c5bcc7" is progressing.
reason: ReplicaSetUpdated
status: 'True'
type: Progressing
observed_generation: 1
ready_replicas: null
replicas: 1
unavailable_replicas: 1
updated_replicas: 1
...


I0623 01:30:57.031983 140564729067264 xds_k8s_testcase.py:244] --- Finished subTest __main__.OutlierDetectionTest.test_outlier_detection.07_start_test_client ---
I0623 01:30:57.032123 140564729067264 xds_k8s_testcase.py:640] ----- TestMethod __main__.OutlierDetectionTest.test_outlier_detection teardown -----
I0623 01:30:57.143256 140564729067264 traffic_director.py:665] Deleting Forwarding rule "psm-interop-forwarding-rule-20230623-0822-7gihn"
I0623 01:30:57.869460 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687509057168-5fec7d1e864db-f91f94f6-fb5ef5ce
I0623 01:31:15.453729 140564729067264 traffic_director.py:704] Deleting alternative Forwarding rule "psm-interop-forwarding-rule-alt-20230623-0822-7gihn"
I0623 01:31:15.597975 140564729067264 compute.py:507] Resource forwardingRule "psm-interop-forwarding-rule-alt-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:31:15.598309 140564729067264 traffic_director.py:596] Deleting HTTP Target proxy "psm-interop-target-proxy-20230623-0822-7gihn"
I0623 01:31:16.061075 140564729067264 compute.py:507] Resource targetHttpProxy "psm-interop-target-proxy-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:31:16.061318 140564729067264 traffic_director.py:584] Deleting Target GRPC proxy "psm-interop-target-proxy-20230623-0822-7gihn"
I0623 01:31:16.908747 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687509076079-5fec7d308f1f2-e1c4fbc8-f8e8007f
I0623 01:31:19.217782 140564729067264 traffic_director.py:624] Deleting alternative Target GRPC proxy "psm-interop-target-proxy-alt-20230623-0822-7gihn"
I0623 01:31:19.542830 140564729067264 compute.py:507] Resource targetGrpcProxy "psm-interop-target-proxy-alt-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:31:19.543079 140564729067264 traffic_director.py:516] Deleting URL Map "psm-interop-url-map-20230623-0822-7gihn"
I0623 01:31:20.241405 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687509079557-5fec7d33e0701-5279e26b-9a1b0946
I0623 01:31:22.573277 140564729067264 traffic_director.py:552] Deleting alternative URL Map "psm-interop-url-map-alt-20230623-0822-7gihn"
I0623 01:31:22.952876 140564729067264 compute.py:507] Resource urlMap "psm-interop-url-map-alt-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:31:22.953123 140564729067264 traffic_director.py:239] Deleting Backend Service "psm-interop-backend-service-20230623-0822-7gihn"
I0623 01:31:23.640466 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687509082968-5fec7d372103f-40d6f419-b9ef07d9
I0623 01:32:05.003178 140564729067264 traffic_director.py:323] Deleting Alternative Backend Service "psm-interop-backend-service-alt-20230623-0822-7gihn"
I0623 01:32:05.426369 140564729067264 compute.py:507] Resource backendService "psm-interop-backend-service-alt-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:32:05.426624 140564729067264 traffic_director.py:396] Deleting Affinity Backend Service "psm-interop-backend-service-affinity-20230623-0822-7gihn"
I0623 01:32:05.814026 140564729067264 compute.py:507] Resource backendService "psm-interop-backend-service-affinity-20230623-0822-7gihn" not deleted since it does not exist
I0623 01:32:05.814346 140564729067264 traffic_director.py:198] Deleting Health Check "psm-interop-health-check-20230623-0822-7gihn"
I0623 01:32:06.386535 140564729067264 compute.py:535] Waiting 600 sec for compute operation id: operation-1687509125932-5fec7d601a51e-0b51145d-b1ef2017
I0623 01:32:10.790899 140564729067264 k8s_base_runner.py:462] Deleting deployment psm-grpc-client
/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/kubernetes/client/rest.py:44: DeprecationWarning: HTTPResponse.getheaders() is deprecated and will be removed in urllib3 v2.1.0. Instead access HTTPResponse.headers directly.
return self.urllib3_response.getheaders()
I0623 01:32:20.855715 140564729067264 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-client-20230623-0822-7gihn/psm-grpc-client] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:32:21.332722 140564729067264 k8s_base_runner.py:487] Deleting service account psm-grpc-client
I0623 01:32:21.405439 140564729067264 k8s_base_runner.py:499] Deleting namespace psm-interop-client-20230623-0822-7gihn
I0623 01:32:51.561884 140564729067264 k8s_base_runner.py:462] Deleting deployment psm-grpc-server
I0623 01:33:01.666483 140564729067264 k8s_base_runner.py:474] Deleting service psm-grpc-server
I0623 01:33:02.749429 140564729067264 k8s_base_runner.py:356] Revoking roles/iam.workloadIdentityUser from serviceAccount:grpc-testing.svc.id.goog[psm-interop-server-20230623-0822-7gihn/psm-grpc-server] for GCP Service Account xds-k8s-interop-tests@grpc-testing.iam.gserviceaccount.com
I0623 01:33:03.108102 140564729067264 k8s_base_runner.py:487] Deleting service account psm-grpc-server
I0623 01:33:03.177018 140564729067264 k8s_base_runner.py:499] Deleting namespace psm-interop-server-20230623-0822-7gihn
I0623 01:33:33.277679 140564729067264 xds_k8s_testcase.py:664] ----- Test client/server logs -----
I0623 01:33:33.278175 140564729067264 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-client-ou0x07j4:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-client%22%0Aresource.labels.namespace_name%3D%22psm-interop-client-20230623-0822-7gihn%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-client-ou0x07j4%22;timeRange=2023-06-23T08:25:53.218617Z%2F2023-06-23T08:32:51.550499Z?project=grpc-testing
I0623 01:33:33.278426 140564729067264 k8s_base_runner.py:672] GCP Logs Explorer link to psm-grpc-server-nhr48cwk:
https://console.cloud.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22grpc-testing%22%0Aresource.labels.container_name%3D%22psm-grpc-server%22%0Aresource.labels.namespace_name%3D%22psm-interop-server-20230623-0822-7gihn%22%0Alabels.%22k8s-pod%2Fdeployment_id%22%3D%22psm-grpc-server-nhr48cwk%22;timeRange=2023-06-23T08:24:28.463045Z%2F2023-06-23T08:33:33.277479Z?project=grpc-testing
[ FAILED ] OutlierDetectionTest.test_outlier_detection
[ FAILED ] OutlierDetectionTest.test_outlier_detection
======================================================================
ERROR: test_outlier_detection (__main__.OutlierDetectionTest)
OutlierDetectionTest.test_outlier_detection
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/tests/outlier_detection_test.py", line 118, in test_outlier_detection
test_client = self.startTestClient(test_servers[0], qps=_QPS)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 776, in startTestClient
test_client = self.client_runner.run(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_xds_client_runner.py", line 168, in run
self._wait_deployment_with_available_replicas(self.deployment_name)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/test_app/runners/k8s/k8s_base_runner.py", line 518, in _wait_deployment_with_available_replicas
self.k8s_namespace.wait_for_deployment_available_replicas(
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/infrastructure/k8s.py", line 476, in wait_for_deployment_available_replicas
retryer(self.get_deployment, name)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 423, in __call__
do = self.iter(retry_state=retry_state)
File "/home/kbuilder/.pyenv/versions/k8s_xds_test_runner/lib/python3.9/site-packages/tenacity/__init__.py", line 369, in iter
return self.retry_error_callback(retry_state=retry_state)
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/helpers/retryers.py", line 141, in error_handler
raise RetryError(
framework.helpers.retryers.RetryError: Retry error calling framework.infrastructure.k8s.KubernetesNamespace.get_deployment: timeout 0:05:00 (h:mm:ss) exceeded. Check result callback returned False.

======================================================================
FAIL: test_outlier_detection (__main__.OutlierDetectionTest)
OutlierDetectionTest.test_outlier_detection
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmpfs/tmp/tmp.OvLKtVktid/grpc/tools/run_tests/xds_k8s_test_driver/framework/xds_k8s_testcase.py", line 669, in tearDown
self.assertEqual(
AssertionError: 5 != 0 : Client pods unexpectedly restarted 5 times during test. In most cases, this is caused by the test client app crash.

----------------------------------------------------------------------
Ran 1 test in 679.851s

FAILED (failures=1, errors=1)
+ (( ++failed_tests ))
+ echo 'Failed test suites: 7'
Failed test suites: 7


[ID: 4659972] Command finished after 5557 secs, exit value: 0


Warning: Permanently added 'localhost' (ED25519) to the list of known hosts.
[01:33:34] Collecting build artifacts from build VM
[01:33:39] Kokoro builder finished